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ABSTRACT 


This thesis explores the top-level requirements for an Automated Tool for 
Acquisition Program Management Students (ATAPMS) that is designed to enhance 
training and education in the acquisition management field. The Department of Defense 
(DoD) has identified the education and training of the acquisition workforce as a strategy 
to help make the acquisition system more effective and efficient. As a result, the DoD 
established the Defense Acquisition University (DAU) to provide the required education 
and training. More recently, EO 13111 and the Defense Reform Initiative have presented 
a mandate for the DoD to find ways to use technology to further this strategy. 

Currently, the consortium schools of the DAU are using emerging technologies to 
increase access to their courses. However, the DAU curricula lack automated acquisition 
management training programs that allow instructors to qualitatively assess students' 
work. 

This thesis recommends a set of top-level requirements for an automated program 
that are in compliance with the Advanced Distance Learning Initiative. It then illustrates 
through a prototype module, using a commercial authoring tool, how an ATAPMS can 
assist the DAU instructors teach the critical aspects of Acquisition Program Management. 
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I. 


INTRODUCTION 


A. AREA OF RESEARCH 

The primary focus of this research is to explore the top-level design requirements 
for an automated education and training program for acquisition management topics. The 
purpose of this automated program is to enhance the training and education in the 
acquisition management field. This thesis recommends a set of top-level requirements 
and then illustrates, by developing a prototype, how an automated program can assist the 
Defense Acquisition University (DAU) consortium instructors teach the critical aspects of 
Acquisition Program Management. 

B. BACKGROUND 

The Department of Defense includes improvement to education and training as 
one of the goals of Acquisition Reform [Ref. 1]. Additionally, Executive Order 13111 
[Ref. 2] directs federal agencies to take certain steps to enhance an employee's training 
opportunities through the use of technology. The DAU, through its 14 consortium schools 
which include the Naval Postgraduate School (NPS), provides education and training for 
acquisition professionals throughout the Department of Defense. Currently, the DAU is 
using emerging technologies to increase access to its courses. However, the DAU 
curricula lack acquisition management training programs that are dynamic or that allow 
instructors to qualitatively assess students' work. 

C. RESEARCH QUESTIONS 

The primary research question addressed in this thesis is: 
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-What are the top-level requirements for developing an automated program that 
assists the DAU instructors with teaching the critical aspects of Acquisition Program 
Management? 

Subsidiary research questions are: 

-What are the existing automated training programs for Acquisition Program 
Management students? 

-What are the critical aspects of Acquisition Program Management? 

-To what extent are they adequately presented in the existing automated training 
tools? 

-What are the needs of the DAU consortium instructors that can be incorporated 
in an automated program? 

-How may these design requirements be implemented in a prototype? 

D. METHODS 

The first objective of this research is to document the critical aspects of 
Acquisition Program Management and provide an overview of the existing automated 
training tools used to teach them. This is accomplished through personal interviews and a 
literature review of sources including: 

-Staff members and instructors at several of the Defense Acquisition University 
consortium schools, including the Naval Postgraduate School. 

-References and publications available at the Naval Postgraduate School library. 
-Internet websites (DoD, academic, and commercial). 

The second objective is determining the top-level requirements for developing an 
automated program that assists the DAU consortium instructors with teaching the critical 
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aspects of Acquisition Program Management. This is also accomplished through personal 
interviews and a literature review of sources including: 

-Instructors at the Naval Postgraduate School. 

-Internet websites (DoD, academic, and commercial). 

After developing these top-level requirements, this thesis illustrates them using a 
prototype of a course module. 

E. ORGANIZATION 

Chapter II (Background) explores the DoD’s efforts to improve the acquisition 
system though the proper training and education of the acquisition workforce. It also 
details the latest developments in expanding the reach and reducing the cost of providing 
education and training through the development of standards and the use of technology. 
Finally, the chapter identifies the need for an application that possess a feedback 
mechanism that enables an instructor to assess a user's opinion or overall understanding 
of concepts. 

Chapter m (Program Management) provides an overview of program 
management using an industry model to present the related functional disciplines that a 
program manager is expected to integrate during the program life cycle. It also introduces 
the concept of an "Automated Tool for Acquisition Program Management Students" 
(ATAPMS) to assist the DAU instructors. 

Chapter IV (Structure and Content Requirements) develops the structure, 
strategies and topics for an ATAPMS tailored to the Naval Postgraduate School 
instructors' needs. 
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Chapter V (Design of the Program Management Training Tool) presents one 
possible approach to devising an ATAPMS application that meets the requirements 
specified in Chapter Four. The chapter enumerates strategies for acquiring the software 
required to support an ATAPMS application and then uses one of those strategies for 
demonstrating the construction of a notional module. 

Chapter VI (Conclusions and Recommendations) summarizes the findings of the 
research and presents recommendations for further research and study. 

F. BENEFITS OF STUDY 

This study provides the requirements and a design for an automated tool that once 
implemented, can be used to enhance the Naval Postgraduate School's capability to 
provide instruction on Acquisition Program Management. This application is also 
relevant to the Defense Acquisition University consortium schools because it allows the 
host institution to offer a portable, comprehensive, and thoroughly relevant automated 
program through which students can demonstrate their knowledge on the fundamental 
aspects of Acquisition Program Management. The program also adheres to the precepts of 
the Defense Reform Initiative, the Advanced Distance Learning Initiative, and Executive 
Order 13111, "Using Technology to Improve Training Opportunities." 



II. BACKGROUND 


A. INTRODUCTION 

This chapter illustrates the need to execute the Defense Acquisition mission as 
effectively and efficiently as possible. It also explores the Department of Defense's (DoD) 
efforts to improve the acquisition system though the proper training and education of the 
acquisition workforce. Finally, this chapter details the latest developments in expanding 
the reach and reducing the cost of providing education and training through the 
development of standards and the use of technology. 

B. MOTIVATION FOR BETTER MANAGEMENT 

The Defense Acquisition mission is extensive and complex and requires a 

significant amount of the DoD's resources. The scope of this acquisition mission covers: 

The conceptualization, initiation, design, development, test, contracting, 
production, fielding, deployment, and logistic support, modification, and 
disposal of weapon and other systems, supplies, or services (including 
construction) to satisfy DoD needs, intended for use in or in support of 
military missions. [Ref. 3] 

Every year the DoD spends approximately $80 billion in Research, Development, Test 
and Evaluation (RDT&E) and Procurement appropriated funds for these acquisition- 
related expenses, or approximately 30% of the overall DoD annual budget [Ref. 4: p. 6]. 

Over the last 15 years, the DoD has experienced a trend of shrinking budgets 
(Figure 2-l)[Ref. 4: p. 89]. While fiscal year 2000 to fiscal year 2004 axe expected to 
provide a slight reversal of the downward trend, the DoD budget as an overall percentage 
of the Federal budget remains historically low (Figure 2-2) [Ref. 5]. With such a 
significant percentage of DoD capital expended in the area of acquisition, especially in 
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view of the current budgetary trend, there is a need to be as effective and efficient as 


possible when acquiring the required goods and services. 
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Figure 2-1. DoD Total Obligation Authority. 



Figure 2-2. Defense Outlay as a Percentage of the Overall Federal Budget. 


C. DAWIA AND DAU 

One method the DoD uses to improve the overall acquisition system is to improve 
the training and education of the people who work in the acquisition field. The standards 
and infrastructure used for the training and education provided today were established in 






1990 through the Defense Acquisition Workforce Improvement Act (DAWLA), Public 
Law 101-501, Title XU The law directed the following [Ref. 6: p. 1638]: 

-Establishment of policies and procedures for acquisition personnel 
education and training programs, including intern, cooperative education, 
and scholarship programs. 

-Authorization of special pay for acquisition officers in certain critical 
positions, and repayment of student loans to facilitate recruitment and 
retention of acquisition personnel. 

-Revision of policies and procedures for the recruitment, training, and 
career development of military and civilian defense acquisition personnel. 

-Establishment of acquisition corps for each military department and for 
other defense agencies. 

-Establishment of the Defense Acquisition University. 

The DoD is implementing the law through the adoption of standards detailed in 
DoD manual 5000.52-M "Acquisition Career Development Program" released in 
November 22, 1995. The manual "provides information, guidance, and requirements for 
improving the management and professionalism of the acquisition workforce" [Ref. 7: p. 
14]. 


As directed by the DAWIA, the DoD created the Defense Acquisition University 
(DAU) in 1990. The DAU is responsible for coordinating the provision of the mandatory, 
assignment specific, and continuing education courses that meet the standards prescribed 
by DoD 5000.52-M for military and civilian personnel serving in several acquisition 
career fields. The DAU's mission is to educate and train professionals for effective 
service in the Defense acquisition system. [Ref. 8] 

The DAU was established as a consortium of 12 DoD educational and training 
institutions located throughout the country. These members provide more than 85 
acquisition courses to the approximate 106,000 entry, intermediate, and senior level 
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civilian and uniformed personnel [Ref. 9: p. 3] to allow them to attain certification in one 

or more of the following 11 defense acquisition career fields [Ref. 9: p. 27]: 

-Acquisition Logistics 
-Auditing 

-Business, Cost Estimating, and Financial Management 

-Communications-Computer Systems 

-Contracting 

-Industrial and /or Contract Property Management 
-Manufacturing and Production 
-Program Management 
-Purchasing 

-Systems Planning, Research, Development and Engineering 
-Test and Evaluation 

As of 1999, the consortium members providing the training and education to the 
acquisition workforce were [Ref. 9: p. 4]: 

-Air Force Institute of Technology (AFIT) 

-Lackland Training Facility 

-Army Logistics Management College (ALMC) 

-Defense Logistics Agency Center for Training, Education and Development 
(formerly DCPSO) 

-Defense Contract Audit Institute (DCAI) 

-Defense Systems Management College (DSMC) 

-Information Resource Management College (IRMC) 

-Industrial College of the Armed Forces (ICAF) 

-Naval Center for Acquisition Training (NCAT) 

-Naval Postgraduate School (NPS) 

-Naval Facilities Contracting Training Center (NFCTC) 

-Naval Warfare Assessment Division (NWAD) 

D. MANDATES FOR TECHNOLOGY 

Although today's acquisition training and education infrastructure was originally 
shaped by the passage of DAWIA, that infrastructure is continuously influenced by 
developments affecting education and training within the Federal government. One of 
these developments is the drive to leverage technology as a way to reduce cost and 
increase the availability of training and education opportunities. 


8 



1. Defense Reform Initiative 

The Secretary of Defense introduced the Defense Reform Initiative (DRI) on 
November 10 th , 1997 to improve business practices in the Department of Defense. The 
DRI is described by the DoD as an effort to coordinate several internal and external 


ongoing reform initiatives and to bring the Department "in tune with the times”. The 
Secretary’s intent was to use the DRI as a mechanism to identify savings and migrate 
resources to support modernization accounts. [Ref. 1] 
a) DRI Goals 

The DRI originally focused on initiatives in four main areas, and 

eventually expanded to include nine additional elements. The initial areas were [Ref. 1]: 

-Reengineer: Adopt modem business practices to achieve world-class 
standards of performance; 

-Consolidate: Streamline organizations to remove redundancy and 
maximize synergy; 

-Compete: Apply market mechanisms to improve quality, reduce costs, 
and respond to customer needs; 

-Streamline: Reduce excess support structures to free resources and focus 
on core competencies. 

The following nine elements are the expanded areas, one of which includes 
Acquisition [Ref. 1]: 

-Adopting Best Business Practices 

-Quality of Life 

-Organizational Streamlining 

-Competition 

-Infrastructure 

-Logistics 

-Cyberspace 

-Homeland Defense 

-Acquisition 
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b) Acquisition Element 

Acquisition Reform supports the DoD's efforts to provide the warfighter 
with products that "work better, cost less, and are obtained from a globally competitive 
national industry base" [Ref. 1]. The major goals of the acquisition reform effort include 
[Ref. 1]: 

-Fostering Partnership: Modernizing defense; partnering with 
communities; decreasing paper transactions; and reducing toxic pollution. 

-Internal Reinvention: Streamlining our workforce; providing effective 
cost accounting; reducing excess inventory; and minimizing weapons cost 
growth. 

-Delivering Great Service: New weapons in less time; better logistic 
supply services; simplifying buying of goods and services; and educating 
and training the defense acquisition workforce. 

The principal controlling documents for Acquisition programs are DoD 
Directive 5000.1; "Defense Acquisition", and DoD Regulation 5000.2-R; "Mandatory 
Procedures for Major Defense Acquisition Programs (MDAPs) and Major Automated 
Information Systems (MAISs) Acquisition Programs." DoDD 5000.1 describes broad 
management principles that are applicable to all DoD acquisition programs. DoD 5000.2- 
R establishes mandatory procedures for MDAPs and MAISs. [Ref. 10] 

As part of the acquisition reform effort, the DoD issued an update to the 5000 
series Directive and Regulation. This update defines an acquisition environment that 
makes DoD "the smartest, most responsive buyer of the best goods and services that meet 
our warfighters' needs, at the best dollar value over the life of the product to help meet 
these efforts" [Ref. 10]. The updated documents contained six major themes that support 
the acquisition reform effort [Ref. 10]: 
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-Teamwork: We must work together as a team to build successful 
programs, identify problems early, and maintain a cooperative spirit of 
resolution, thereby providing programs the highest opportunity for success. 
-Tailoring: Milestone Decision Authorities (MDA should strive to tailor 
most aspects of the acquisition process, including program documentation, 
acquisition phases, and the timing, scope, and level of decision reviews. 
-Empowerment: Program managers do not have to ask permission to take 
actions that are otherwise permitted by law and are within the scope of 
their charters. The Department has long relied on volumes of guidance and 
regulation, prescribing every detail of both process and documentation. 

DoD has also had a habit of dealing with industry through a rigid system 
of military specifications. DoDD 5000.1 and DoD 5000.2-R reflect current 
efforts to empower our people and our vendors to do the best they can. 

-Cost as an Independent Variable (CAIV): The acquisition process 
described in DoDD 5000.1 and DoD 5000.2-R must consider both 
performance requirements and fiscal constraints. Accordingly, cost must 
also be an independent variable in programmatic decisions, with 
responsible cost objectives set for each program phase. 

-Commercial Products: Historically, DoD has relied on segments of the 
U.S. technology and industrial base principally dedicated to supporting 
DoD requirements...Acquisition of commercial items, components, 
processes, and practices provides rapid and affordable application of these 
technologies to validated, DoD mission needs. 

-Best Practices: DoD 5000.2-R describes a simplified and flexible 
management process, modeled on sound business practices. Acquisitions 
of the future must take into account customary commercial practices in 
developing acquisition strategies and contracting arrangements. 

"Educating and training the workforce" is a subset to one of the major goals of the 

Acquisition element. In an effort to support the acquisition reform education and training 

goal, the DAU is planning to use emerging technologies to increase access to its courses. 

Further, in its "Implementation Plan for Technology Based Education and Training" 

document, the DAU recognizes that these courses need to be "designed and delivered so 

they are updated quickly and inexpensively and their shelf life can be extended." [Ref. 11] 

Currently, the DAU is working toward these efforts and taking advantage of new 

distributed learning technology to modernize its curriculum. In 1997, 10% of the DAU's 

courses were modernized and delivered by distributed learning technologies. The goal is 
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to complete modernization of all of its courses through technology enhancements by 
2003. [Ref. 12] 

2. Executive Order 13111 

President Clinton signed Executive Order (EO) 13111, "Using Technology to 
Improve Tr ainin g Opportunities", on 12 January 1999. This EO directs federal agencies to 
take certain steps to enhance an employee's training opportunities through the use of 
technology. Specifically, the Order states "A coordinated federal effort is needed to 
provide flexible training opportunities to employees and to explore how federal training 
programs, initiatives, and policies can better support lifelong learning through the use of 
learning technology." To assist with this goal, the EO established the President's Task 
Force on federal training technology and created an advisory committee on expanding 
training opportunities. [Ref. 2] 

3. Advanced Distance Learning Initiative 

The DoD launched the Advanced Distance Learning (ADL) Initiative in 
November of 1997. The ADL is an effort by the DoD to identify more efficient and 
effective ways to educate and train the DoD personnel through the development of 
advanced distributed learning technologies. The purpose of the ADL initiative is to 
"ensure access to high-quality education and training materials that can be tailored to 
individual learner needs and can be made available whenever and wherever they are 
required." [Ref. 13] 

In his November 1998 memorandum "Developing and Implementing the DoD 
Advanced Distributed Learning (ADL) Initiative", the Deputy Secretary of Defense 
directed the Under Secretary of Defense for Personnel and Readiness (USD(P&R)) to 
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lead the ADL Initiative and to work with the Services, Joint Staff, and other DoD 
components in developing and implementing ADL technologies across the DoD [Ref. 
14]. Through this initiative, the DoD seeks to accelerate large-scale development of 
"dynamic and cost effective learning software...in order to meet the education and 
training needs of the military and the nation's workforce in the 21 st century." [Ref. 13] 

a) Lack of Standards 

This initiative addresses the problem created by a lack of standards within 
the education software industry. Developers of online learning resources have many 
different software tools to choose from to create their courses. Prior to the ADL, however, 
there was no common learning management software reference model. As a result, the 
available computer-based training authoring tools have historically been developed on a 
proprietary, company by company basis. This scenario results in high development costs 
and limits the reuse potential for already developed software. [Ref. 15] 

b) Instructional Management Systems Project 

To help alleviate the commonality problem within the DoD and 
accomplish the previously stated DoD objectives, the DoD has teamed with the 
Instructional Management System (IMS) Project. The IMS is a consortium of government 
organizations, over 1,600 colleges and universities, and 150 corporations [Ref. 13]. The 
IMS project is an effort to develop next-generation open architecture for online learning 
software. 

Overall, the IMS attempts to address three obstacles for providing 
effective online materials and learning environments [Ref. 16]: 
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-Lack of standards for locating and operating interactive platform- 
independent materials 

-Lack of support for the collaborative and dynamic nature of learning 
-Lack of incentives and structure for developing and sharing content 

By teaming with the IMS consortium, the DoD is helping to formulate industry - 

wide voluntary learning specification guidelines that will make learning software 

interoperable, reusable, and accessible through the Internet [Ref. 15]. The IMS technical 

specifications will provide the general guidelines and requirements developers must write 

to for creating interoperable content and management systems [Ref. 17]. 

The IMS efforts are centered on specifications in four areas [Ref. 17]: 

-Meta-data: Descriptive labels can be used to index learning resources to 
make them easier to find and use. Such labels are "data about data" and 
are referred to as "meta-data." An example of meta-data is the label on a 
can of soup, which describes the can’s ingredients, weight, cost, and so 
forth...A meta-data specification makes the process of finding and using a 
resource more efficient by providing a structure of defined elements that 
describe, or catalog, the learning resource, along with requirements about 
how the elements are to be used and represented. [Ref. 18] 

-Packaging and Run-time services: This area focuses on specifications for 
searching, digital packaging, and modularizing content with the goal that 
any content can run on any learning server. Resources can be drawn from 
any developer and potentially reused in different learning activities. [Ref. 

19] 

-Profiles: User profiles are mobile, user-controlled collections of personal 
and educational data including personal information, performance 
information, and preference information. This data represents a rich 
resource that a user can draw on to facilitate, customize, and manage his or 
her learning experience(s). [Ref. 19] 

-Enterprise integration: The IMS Enterprise Information Model describes 
data structures that are used to provide interoperability of Internet-based 
Instructional Management systems with other Enterprise systems used to 
support the operations of an organization. The objective of the IMS 
Enterprise Information Model is to define a standardized set of structures 
that can be used to exchange data between different systems. These 
structures provide the basis for standardized data bindings that allow 
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software developers and implementers to create Instructional Management 

processes that interoperate across systems developed independently by 

various software developers. [Ref. 20] 

E. CURRENT METHODS USED TO ASSIST ACQUISITION TRAINING 

The DAU is using technology-based education and training methods to maintain 
pace with the reforms in the acquisition process. While the DAU has successfully 
converted several of its courses to a technology-based delivery, the methods for 
presenting the acquisition related training and education are still as diverse as the 
consortium members providing it [Ref. 9: p i]. A survey by the author of the delivery 
medium used by the consortium members revealed several methods of course 
presentation including traditional classroom, mobile teaching, video teleconferencing, 
automated programs, and the Internet. 

1. Classroom 

Classroom training is the traditional method of providing instruction. Students 
typically travel to the institution's location from several dispersed areas to receive training 
and instruction. 

2. Mobile Teaching 

One of the non-traditional methods of providing instruction to the students is 
mobile teaching. Using the mobile teaching concept, the instructor travels to the student's 
location instead of the students traveling to the instructor's location. 

3. Video Teleconference Courses 

Several consortium members conduct courses using video teleconference 
technology as the medium. The typical scenario consists of an instructor co-located at one 
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location with a small contingency of students, while students communicate with the 
instructor via real-time voice and video from remote locations. 

4. Non-interactive Automated Courses 

Some of the consortium members offer course material in a non-interactive 
automated format. One member provides a laptop computer that has course material 
preloaded. The course material is presented in a dynamic, hypertext-linked environment. 
This program is not interactive, however, as its purpose is to present information instead 
of soliciting information from the user. 

The NPS uses a program called Systems Acquisition For Executives (SAFE) that 
works on a slightly different premise. Instead of automated course material, SAFE users 
only have access to a workbook. The instructor maintains access to the actual SAFE 
program and provides the students with the results of the SAFE program simulations. 

SAFE is a computer-assisted exercise that allows the user to influence the 
outcome of the acquisition program in the scenario based on decisions concerning the 
cost, schedule, and performance parameters. The scenario and worksheets are contained 
in a manual and the students use this information to provide the instructor with their cost, 
schedule, and performance decisions at discrete points through the exercise. When the 
instructor enters the students' data into the automated program, the SAFE program 
simulates an outcome and then generates a response indicating the updated status of each 
student's program. The students begin another session of decision-making based on the 
updated status of their program and the next set of alternatives presented by the scenario 
in the workbook. 
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5. Interactive Internet Courses 


The DAU provides several courses using the Internet or a compact disk as the 
medium. Each course contains several lessons that are further organized by topics. These 
courses are self-paced, interactive modules. They contain hypertext-links that allow 
students to navigate through instructional material and case scenarios. The interactive 
multi-media capable modules require students to input information gleaned from the 
instructional text and case scenarios. The modules provide the user feedback based on the 
information entered. All of this information is viewed on the user's web browser. These 
courses typically have associated tests delivered in electronic format that students are 
required to pass before earning credit for completing the course. 

F. CURRENT DEFICIENCIES 

The intent of the Executive Order, the DRI, and the ADL initiative is clear: the 
DAU must leverage the advances of technology and incorporate automation and distance 
learning into the training and education of the acquisition workforce to highest degree 
possible. 

Even though the DAU has a plan for achieving this objective and the number of 
interactive automated courses continue to grow, the current courses lack the ability to 
provide education and training on a qualitative basis. The responses from the interactive 
courses are predetermined and codified in the software. There is no interaction with a 
"subject matter expert" to review the student's input and provide a qualitative response for 
scenarios that have more than one correct answer. In essence, while courses do interact 
with the user, they do not possess a feedback mechanism to assess a user's opinion or 
overall understanding of concepts. 
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G. CHAPTER SUMMARY 

Because the acquisition mission is broad and consumes a significant percentage of 
the DoD's resources, there is an unquestionable need to execute that mission in the most 
effective and efficient manner as possible. Accordingly, the DoD has identified the proper 
education and training of the acquisition workforce as a strategy to help improve the 
acquisition system. As directed by the DAWIA, the DOD established the DAU to provide 
this required education and training. 

There is a current emphasis on leveraging the use of technology to broaden the 
accessibility and reduce the cost of providing the required education and training to the 
acquisition workforce. EO 13111 and the DRI provide a clear mandate for the DoD to 
find ways to use technology to further this goal. Additionally, the efforts that the DoD 
have undertaken in the ADL initiative will help provide the mechanism for the added 
reach and the decreased cost through the creation of industry-wide standards for 
educational software. 

Even though the DAU is responding to this mandate and providing interactive 
distance learning courses, there is one type of course not yet offered. There are currently 
no automated courses that allow for qualitative assessments of a student's work. 

The next chapter provides an overview of one of the critical acquisition career 
fields: program management. Because program management is a diverse field that 
integrates the other disciplines within acquisition, this thesis will use it as the subject of 
an example course in a subsequent chapter. 



ffl. PROGRAM MANAGEMENT 


A. INTRODUCTION 

This chapter provides an overview of program management. The chapter uses a 
model borrowed from industry to present the related functional disciplines that a program 
manager is expected to integrate during the program life cycle. The disciplines presented 
in this chapter will be linked to their Defense Acquisition counterparts in the following 
chapter. 

B. IMPORTANCE OF PROGRAM MANAGEMENT 

Program management is listed as one of the 11 critical career fields within the 
acquisition system. Specifically, the DoD defines program management as [Ref. 21: p. 
12 ]: 

The process whereby a single leader exercises centralized authority 
and responsibility for planning, organizing, staffing, controlling, and 
leading the combined efforts of participating/assigned civilian and military 
personnel and organizations, for the management of a specific defense 
acquisition program or programs, through development, production, 
deployment, operations, support, and disposal. 

Program management is critical to the acquisition system because it provides a 
single point of contact who acts as an "integrator" of a complex system of "differing but 
related functional disciplines" [Ref. 21: p. 13]. These functional disciplines relate to areas 
such as business and financial management, logistics, systems engineering, software 
management, test and evaluation, manufacturing management and others [Ref. 21: p. 13]. 
To provide a more detailed understanding of program management, the remainder of this 
chapter is dedicated to presenting an overview of program management and its related 
functional disciplines. 
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C. PROGRAM MANAGEMENT OVERVIEW 

Industry typically uses a program management model similar to that used in the 
DoD acquisition system. The contractor staff usually operates its program office to 
parallel the military office it is supporting on a defense project [Ref. 21: p. 14]. 

Because the industry and defense models of program management are similar, this 
chapter uses an industry model from Forsberg, Mooz, and Cotterman's textbook 
"Visualizing Project Management" as a reference to describe the "differing but related 
functional disciplines" that a program manager must integrate. 

While the program management models are similar, the terminology is not always 
the same. The DoD uses the term "program manager" and industry commonly uses the 
term "project manager". Even though the two terms are synonymous [Ref. 21: p. 12], the 
remainder of this section uses the term "project manager" because that is the term used by 
the authors. 

In their textbook, Forsberg, Mooz, and Cotterman divide the project management 
process into four essentials: Common Vocabulary, Teamwork, Project Cycle, and the 
Project Management Elements [Ref. 22: p. 27], The first two components, Common 
Vocabulary and Teamwork, are characterized as perpetual properties. They are always 
present throughout the lifecycle of a project. The third component. Project Cycle, relates 
to sequentially occurring activities of a project. Finally, the Project Management 
Elements are described as situational and are applied on an as needed basis. 

1. Common Vocabulary 

A common vocabulary is the first perpetual property of projects. The authors 
contend that to succeed at project management, one must communicate using clearly 
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defined terms. Otherwise, vocabulary problems can lead to conflict and ultimately destroy 
teamwork. Therefore, a common vocabulary is necessary before you can develop 
teamwork. 

This is true in part because problems occur when terms used in project 
management are confused with similar terms used in the context of other fields. An 
example given is the contractor who mistook the term "prototype" to mean an engineering 
model that could be based on commercially available parts instead of a model built to 
released drawings with production-worthy parts and processes [Ref. 22: p. 53]. 

The authors provide many reasons to demonstrate the need for a common 
vocabulary at the project level [Ref. 22: p. 27]: 

-Most companies do not have a common vocabulary. 

-Schools do not have, and consequently do not teach a common vocabulary. 

-Words are used differently across projects, companies, and industries. 

-Terminology manuals, when they exist, are often imprecise. 

-There is little effort to fix the problem. 

To alleviate this deficiency, the authors recommend a terminology manual for 
each project. A terminology manual, taking into consideration the terminology 
appropriate to the industry, company, and the specific project, is indispensable for 
mitigating the communication problems caused by a vocabulary breakdown. 

2. Teamwork 

Teamwork is the second perpetual property of projects. Without a commitment to 
teamwork, project activity is subject to seemingly chaotic activities. In fact, the DoD 
recognized this "essential" by including it as a major theme in the recent update to the 
5000 series documents. 
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To help project managers achieve real teamwork; the authors provide five 
fundamentals of teamwork and the techniques for achieving them [Ref. 22: p. 31]: 

a) Common Goals 

The first step to building teamwork is to clearly define the group 
objectives. Managers need to outline the roles and responsibilities required by the team 
members to accomplish the objectives. Additionally, managers must ensure that team 
members can articulate these common goals and that they are committed to achieving 
them. [Ref. 22: p. 62] 

b) Acknowledged Interdependency and Mutual Respect 

When interdependencies are not acknowledged, there is no footing for 
teamwork. There is only the foundation for "well structured independent effort." [Ref. 22: 
p. 63] For interdependencies to be recognized, the team members need to accept, and 
respect, the roles that must be filled by each team member. The authors list the three steps 
to acknowledging interdependencies and establishing expectations as [Ref. 22: p. 63]: 

-Defining the specific functions, tasks, and individual 
responsibilities. 

-Developing an organizational structure and defining team 
interdependencies. 

-Defining the scope of each team member. 

These steps are not only recommended for use during the initial formation 
of the team, but also for the subsequent inclusion of new team members. 

c) A Common Code of Conduct 

The most commonly overlooked factor in teamwork, according to the 
authors, is a common code of conduct. Even though the obvious conduct issues are 
usually well documented by company and government policies, they still may not be well 
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known to all the team members. Gray areas are especially subject to misinterpretation and 
confusion. Compounding the problem, managers often assume that a code of conduct is 
implied and understood even if it has not be specifically defined and agreed to. The end 
result can be the destruction of teamwork through tension and separation among the team 
members. 


The authors provide guidelines for a code of conduct. To be effective, a 
common code of conduct must [Ref. 22: p. 65]: 

-Establish rules of behavior. 

-Reach consensus on the definition of an ethical code of conduct. 

-Document the most significant factors. 

Managers therefore have a responsibility to review the issues to ensure that 
all team members are aware of the potential problems. Figure 3-1 illustrates an overview 
of legal conduct issues. 

Ethical issues are more difficult to enumerate than legal issues. Individuals 
usually rely on personal values to navigate through conflicts between company practices, 
laws and regulations, and management directives. To help team members negotiate 
ethical issues; the authors provide these general guidelines [Ref. 22: p. 65]: 

-Seek higher management guidance to confirm difficult choices for 
conflicts among the various codes of conduct. 

-If asked to operate in a potentially improper manner, make sure 
that the request is written and verify it with the cognizant 
authority. 

-Report any improper conduct, anonymously if necessary. 
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Peddling 


Indirect 


Figure 3-1 From [Ref. 22: p 65], Legal Conduct Issues. 

d) Shared Rewards 

The concept of shared rewards is simple: effective rewards start with fair 
and equitable compensation for each position on the team. Individual members or the 
entire team can earn rewards. If the project manager prefers to divide cash awards among 
the team members, the authors recommend the money be given in small amounts so more 
can share. Another strategy offered is to simply spend the money on team recognition. 

e) Team Spirit and Energy 

This teamwork fundamental is a factor of personal attitude as well as 
company culture. The authors are careful to point out [Ref. 22: p. 67] that putting the 
team ahead of oneself does not mean that team spirit requires the elimination of 
independent "pace setters." The key for members with driven personalities is to allow 
them to exercise their assertiveness and energy without dominating teammates. 

f) Techniques for Creating Teamwork 

It is the responsibility of the project manager to draw on the strength of the 
team and to neutralize its weakness. To help prevent the manager from micro managing 
or providing too much freedom to the team, the authors present a list of responsibilities 
for the manager. Accordingly, the project manager must [Ref. 22: p. 67]: 
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-Clearly define unambiguous responsibilities. 

-Define and communicate a project process and style. 

-Delegate whenever possible. 

-Empower the team to be accountable. 

-Balance support with direction as required. 

-Train the team, by example, to operate as a team. 

-Deal with under-performers who drag the team down. 

-Establish team-effort rewards. 

-Design the tasks and work packages in a way to encourage 
teamwork. 

Finally, managers need to confirm that their leadership is effective by 
measuring observable behavior. There are positive and negative indicators that a manager 
can observe to gauge the status of the team cohesiveness (Table 3.1) [Ref. 22: p. 73]. 


Positive Indicators 

Negative Indicators 

A positive cooperative climate prevails. 

A climate of suspicion and mistrust exists. 

Information flows freely between team members. 

Information is hoarded or withheld. 

No work is considered beyond an individual's job 
description. 

Fingerpointing and defensiveness prevails. 

Interpersonal interactions are spontaneous and 
positive. 

Counterproductive subgroups and cliques begin to 
form. 

The collective energy of the team is high. 

Fear of failure causes individuals to avoid or 
postpone making important decisions. 


Table 3-1. Positive and Negative Indicators. 


3. Project Cycle 

All projects pass through a sequence of events that the authors identify [Ref. 22: 
p. 32] as the "project cycle". Different organizations have their preferred project models 
from which they tailor to fit each specific project. The project cycle is used as a roadmap 
for the project to progress from "stake-to-stake", or from one milestone to the next. 

Without the use of a field-tested, clearly defined sequence of actions for a project 
to follow, teams are forced to create their own "path". As a result, the consequences for 
not using a proactive framework for the project can be devastating. [Ref. 22: p. 76] 
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The project cycle template described by the authors is decomposed into many 
components and subcomponents. It has Periods, Phases within each Period, Control 
Gates, Activities, and Products. It also contains three “interwoven” layers, the Technical, 
Business, and Budget aspects that occur in parallel within the project cycle. [Ref. 22: p. 
32] 

a) Periods 

The project template is divided into three periods: the Study Period, the 
Implementation Period, and the Operations Period (Figure 3-2). Each of these periods 
corresponds to the three major stages of a project as it progresses from the identification 
of the user's need, through concept determination, implementation, and finally to 
production and user operation. [Ref. 22: p. 77] 

The study period is crucial to the project because it is during this time that the 
scope and funding is determined. Even though a project team must engage in 
considerable analysis and negotiation during the Study Period, a thorough study can often 
prevent lost time and wasted funds as the result of rework. 

The Implementation period sets the contractual foundation for the project. 
Objectives for this period include choosing the highest value bidder, designing and 
building the first article, and testing and verifying the article in accordance with all 
specifications. 

It is during the Operations Period that the user’s needs are fulfilled and the project 
solution in achieved. In a government project the objectives for this period are: to transfer 
the system to the operational location and establish full operational capability, operating 
and maintainin g the system according to the user’s requirements, and identifying system 
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improvements for future use [Ref. 22: p. 82], In commercial projects the system is 
delivered to users in the marketplace. Finally, in applicable projects, planning for 
deactivation takes place. 


The Project Cycle contains Each Period contains 



Figure 3-2 From [Ref. 22: p 33]. The Sequential Project Cycle. 


b) Control Gates 

The authors define a control gate as "a management event important 
enough to be defined and included to the schedule by executive management." [Ref. 22: 
p. 83] They are the major decision points in a project cycle that ensure the scheduled 
activities are completed successfully before allowing the pursuit of new activities. The 
primary objectives of Control Gates are listed as [Ref. 22: p. 83]: 

-To ensure that all current phase activities and products are 
complete. 

-To ensure that progression to the next set of activities is based on 
hard evidence that the team is prepared and that the risk of 
proceeding is acceptable. 

-To promote a synergistic team approach. 


27 









c) Activities 

Activities are those specific actions taken by the project team to meet the 
goals of the project. Examples of these activities include [Ref. 22: p. 34]: 

-Defining user requirements. 

-Comparing trade-off candidate concepts. 

-Developing a user validation approach. 

d) Products 

Products are the output of activities. Products usually need to be approved 
at the Control Gates. Examples of Products include [Ref. 22: p. 34]: 

-System Concept Document. 

-Specifications, drawings, and manuals. 

-Internal hardware and software feasibility models. 

-Deliverable hardware, software, and documentation. 

e) Budget 

The budget aspect is one of three layers to every project cycle and 
represents the activities and events needed to provide the project with the necessary funds 
throughout its cycle. The budget contains all events pertaining to the executive 
management appropriation of project funds and a project manager's securing of funds. 
[Ref. 22: p. 85] 

f) Business 

The business aspect contains all the events related to the overall business 
management of the project and associated contract management. One important business 
management event is pursuing and managing customers. Other events include activities 
needed to solicit, select, and manage the vendors who are participating in the project. 
[Ref. 22: p. 86] 
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g) Technical 

The combination of the events from the three project cycle aspects 
constitute the total cycle (Figure 3-3). The budget and business activities are overlaid on 
the technical aspect to form the complete project cycle (Figure 3-4). Like the other two 
layers, the technical aspect of the project cycle contains its own set of events even though 
it occurs in parallel with the other two aspects. [Ref. 22: p. 33] 

The technical cycle contains all the events related to satisfying the 
technical and quality requirements [Ref. 22: p. 35]. It identifies the activities and events 
from a system engineering perspective required to provide the most effective and efficient 
technical solution to the project requirements. 

The actual process is best visualized as “Vee-Model” rather than in a 
horizontal format (Figure 3-5). The schedule maturity and time progress from left to right 
on the model. On the left side, one performs decomposition and definition while moving 
from system requirements and concepts down to fabrication, assembly, and code to 
"build-to" documentation. The right side of the Vee corresponds to an upward movement, 
aggregating the system's components at successively higher levels while performing 
integration and verification. [Ref. 22: p. 34] 
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The PROJECT CYCLE 



Each event or product of an event, is assigned to the appropriate category 


These events involve planning for, and securing, 
project funding to fuel the project through the cycle. 


Control 

Gates 


Specific actions taken to meet the goafs of the project, 
e.g., Define user requirements. 

Trade-off candidate concepts, 

Develop user validation approach. 


The output of activities - to be approved at the Control Gate, 
e.g.. System Concept Document 

Specifications, drawings, and manuals 

Internal hardware and software feasibility models 

Deliverable hardware, software, and documentation. 


Predetermined decision check points to be satisfied 
before advancing to the next set of activities, 
e.g.. System Concept Review. 



Figure 3-3 From [Ref. 22: p. 34]. Total Cycle Format. 


Technical - contains all events related to satisfying 
the technical and quality requirements. 
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Figure 3-5 From [Ref. 22: p. 108]. The Vee Model. 

The utility of the Vee format is in the capability to better visualize the 
direct correlation between the activities on the left and the right side of the model. One 
example of this correlation is the manner in which the model minimizes the chances that 
requirements are specified in a way that cannot be measured or verified. While using the 
Vee-model, the method of verification to be used on the right side must first be 
determined during activities on the left side before descending down to the next level on 
the left side. 

One of the valuable uses for the Vee-model is as a tool for controlling the 
systems engineering process, which the authors define [Ref. 22: p. 37] as the application 
of the "Systems Analysis and Design" and the "Systems Integration and Verification" 
subprocesses to the technical aspect of the project cycle. These two subprocesses are 
further defined in the following Project Management Elements section. 

Another valuable use for the Vee-model is as a tool for risk management. 
The Vee-model offered by the authors encourages detailed work early in the cycle to 
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reduce risk. An example [Ref. 22: p. 88] is the accommodation of hardware and software 
"requirements-understanding" and technical feasibility models in the first period. Risk 
identification and reduction modeling continues through the conduct of "off-core" 
activities (Figure 3-6). 

Unlike decisions made during the off-core studies, only decisions made at 
the core of the Vee are put under configuration management at the appropriate control 
gates. These configurations comprise the baseline definition for the system under 
development. 


“Off-Core” User Discussions 
and Approvals 

“Are the proposed solutions acceptable?” 


Baseline Being 

^ \ Verification / ^ 

Planned 

Considered 

^ “How to prove that the solution ^ 

Verification 




Baseline 


Core of the “Vee’ 

Plans. Specifications, and 
Products Are Under Project 
Configuration Management 


V 



Baselines to be 
Considered 

{Core of the ”Vee”) 


Baselines to be 
Verified 

{Core of the ‘Vee”) 


“Off-Core” Risk and 
Opportunity Management 
Investigations and Actions 

How are the risks and opportunities of the 
proposed solutions being managed?” 


Time and Baseline Maturity 


Figure 3-6 From [Ref. 22: p. 89]. Vee Model and Off Core Activity. 


4. Project Management Elements 

As the authors note, the technical, schedule and cost performance elements of a 
project do not complement each other. Rather, they are opposing forces that require a 
skillful blending to reach the best solution for the given situation. The Project 
Management Elements, applied situationally throughout the project cycle, provide the 
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tools and techniques needed to keep the aforementioned parameters in balance. [Ref. 22: 
P-39] 


The Project Management Elements consist of 10 categories of management 
responsibilities, functions, techniques, and tools that help manage [Ref. 22: p. 104]: 

-All types of projects. 

-All phases of the Project Cycle. 

-All organizations participating in the project. 

a) Project Requirements 

The project requirements constitute requirement creation and management. 
The major documents created to support this element are the products referred to as 
"outputs" in the project cycle. The framework for applying this element is embodied by 
the Vee-model discussed in the previous section. 

The first technique for determining requirements is the System Analysis 
and Design (SA&D) process. It is applied at every level of the decomposition and 
definition sequence while working down the left side of the Vee-model. The SA&D 
process is initiated by the higher level requirements and constrained by the approved 
baselines. There are several steps within the SA&D (Figure 3-7) [Ref. 22: p. 108]: 

-Define the problem and evaluation criteria. 

-Understand the context of implementation. 

-Define the required behavior and performance. 

-Develop candidate physical or logical solutions. 

-Select the best solution. 

-Get customer approval of solution and add to baseline. 

The Systems Integration and Verification (SI&V) process is the 
counterpart to SA&D. It is an iterative process that is repeated throughout the integration 
and verification sequence while navigating up the right side of the Vee-model format. 
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SI&V provides the framework for using the verification criteria developed during the 
decomposition and definition process while inspecting and testing the systems integration 
that occurs at each level. Similar to SA&D, there are also several steps within the SI&V 
process (Figure 3-8) [Ref. 22: p. 116]: 

-Inspect and test to verification requirements. 

-Integrate with the next configuration item and repeat verification process. 

-If needed, identify and correct deficiencies. 

-If needed, document uncorrectable deficiencies. 

-If needed, confirm impact of uncorrectable deficiencies. 

-If needed, modify approved technical baseline to incorporate deviation. 

Another requirements management responsibility is the "traceability" and 
"accountability" of project requirements. The purpose is to ensure that all requirements 
have been incorporated into the design and that requirement satisfaction is verified by 
test, inspection, demonstration, and if needed, analysis. [Ref. 22: p. 118] 
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Figure 3-7 From [Ref. 22: p. 110]. System Analysis and Design Flow Chart. 
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Verification Requirements and 



Figure 3-8 From [Ref. 22: p. 116]. System Integration and Verification Flow Chart. 


b) Organization Options 

The proper organizational structure can contribute significantly to the 
project's performance and efficiency. Conversely, over-complexity or redundancy in 
organizational design can lead to confused responsibility and inefficiency. To find the 
best organizational fit, it is important to ensure that project requirements are considered 
because each project presents a unique set of organizational requirements. The authors 
present several organizational structure options and the relative strengths and weaknesses 
for each: [Ref. 22: p. 128] 

-Pure functional: This organization is rated as the best for a single 
project that is relatively independent in interface or technology. It 
is rated as not good for any organizations that are managing 
multiple projects. 

-Pure project: This option is rated as the best choice for projects 
that prioritize the schedule or cost performance as high and 
development costs as relatively unimportant. 

-Conventional matrix: This organization configuration works best 
if the project manager controls the funds and has well defined 
interrelationships with the supporting managers. However, the 
matrix fails in those circumstances where the project manager is 
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viewed as a "coordinator" and the support managers operate on a 
"best effort" basis. 

-Collocated Matrix: Managers should consider this option for very 
high priority projects that depend on critical resources and 
technologies, and when on-going involvement with company 
strategy is secondary. 

c) Project Team 

Team formation is a situational process that is ongoing throughout the 

project life-cycle. This is because the process of forming the team involves more than just 

determining staffing; it also encompasses the definition and management of interfaces 

with project stakeholders such as supporting organizations, contractors, upper 

management, and the customer. [Ref. 22: p. 132] 

As a part of the team staffing strategy, senior management should 

document job responsibilities prior to selecting the project manager. The authors state 

[Ref. 22: p. 134] that this documentation should establish responsibility for: 

-Establishing the team and teamwork environment. 

-Inspiring and motivating the team. 

-Ensuring all project requirements are clearly defined and 
documented and disseminated to the lowest level. 

-Ensuring the controls are in place and effective. 

-Controlling baseline requirements through a change control 
system. 

-Ensuring that the visibility techniques are in place and effective. 
-Determining the frequency and the level of status that is 
appropriate for the project. 

-Timely execution of corrective action to recover the project plan. 

The project manager is the key person on the project team. As the authors 
have noted [Ref. 22: p. 135], the project manager must interface in three different areas. 
He or she must meet the requirements of the customer, answer to senior management, and 
provide a positive work environment for the project team. Therefore, selecting the project 
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manager is a critical matchmaking task for upper management that is crucial to the 
project's success. The authors provide [Ref. 22: p. 136] a sample "competency model" 
that can be used when selecting a team member as an objective basis for evaluating the 
key competency factors required for a particular project. Figure 3-9 depicts a model that 
is tailored for selecting a project manager. 

The project manager needs to determine the required functions and related 
skills as shown in Figure 3-9 before selecting team members. For this task, the nature of 
the project will guide the project manager. It is also important to select the core team 
from people who have previous experience at the task or project management level. 


Rating Factor 

Basic 

Score 

Advanced 

Score 

Expert 

Score 

Project 

Management 

Training 

Has had some proj¬ 
ect management 
training 


Has had the com¬ 
pany's or equivalent 
project manage¬ 
ment training 


Has had the com¬ 
pany's, PMI 1 , or 
equivalent certifica¬ 
tion in project man¬ 
agement 


Project 

Management 

Experience 

Has served as a 
deputy or assistant 
project manager 


Has been a success¬ 
ful project manager 


Has managed 
several successful 
programs 


Contracting 

and 

Negotiating 

Is knowledgeable of 
types and applica¬ 
tions of the relevant 
contract types 


Has participated in 
developing contract 
negotiation 
strategies 


Has considerable ex¬ 
perience in contract 
negotiation strategy 
and participating in 
negotiations. 


Sub¬ 

contracting 

Is knowledgeable in 
the difference be¬ 
tween purchasing 
and subcontracting 


Has participated in 
the selection and 
award of subcon¬ 
tracts 


Has successfully 
managed subcon¬ 
tractors 


Decision 

Analysis 

Is aware of the im¬ 
portance and prac¬ 
tice of Analytical 
Decision Analysis 2 


Has been trained in 
Analytical Decision 
Analysis 2 


Has been trained 
and routinely prac¬ 
tices Analytical De¬ 
cision Analysis 2 



Figure 3-9 From [Ref. 22: p. 136]. Example Competency Model. 


In addition to the project manager, there are two other important positions 
that make up the core of the team: the systems engineer and the business manager. The 
systems engineer is responsible for the project's technical integrity while meeting the cost 
and performance objectives. The business manager is responsible for all the business 
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aspects of the project to include planning, scheduling, and contract matters. [Ref. 22: p. 
139] 


d) Project Planning 

The authors define planning as "the process which determines beforehand 
the activities necessary to complete the project." [Ref. 22: p. 148] At the total project 
level, planning is performed in each project cycle phase. As a result, the overall plan is 
composed of separate plans for each period and phase of the project (Figure 3-10). At the 
most basic level, the plan contains [Ref. 22: p. 149]: 

-What is to be done. 

-When it should be done. 

-Who is responsible for doing it 



Figure 3-10 From [Ref. 22: p. 149]. The Total Project Plan. 

The goal of planning parallels the goal of the project: ensure that the 
commitments made to the customer are met. To determine the project deliverables, and 
provide a description of each, the authors recommend [Ref. 22: p. 155] that project 
managers use the Project Product List (PPL) and Fact Sheets. The PPL and Fact Sheets 








are derived from system decomposition and definition and are a listing of all contract 
deliverables, and their quantities required. 

The Work Breakdown Structure (WBS) is a method used to decompose a 
system by assemblies, subassemblies, and components instead of by functional 
organization or discipline (Figure 3-11). The WBS is essential for project planning 
because it becomes the basis for work assignments, budgeting, scheduling, risk 
assessment, cost collection, and performance statusing. The authors offer the following 
guidance when preparing a WBS for hardware or software related project [Ref. 22: p. 
158]: 


-Structure the WBS by product and elements of the product. 

-Include all authorized tasks. 

-Place cost collection one level below budget performance report 
to facilitate problem cause identification. 

-Ensure identifiers for like tasks are similar. 

-Collect all tasks for an element with the element identifier. 

-WBS depth depends on the risk to be managed and reported. 

The WBS provides an essential foundation for the project network and for 
scheduling. To create this foundation, the tasks devised from the WBS are combined to 
form the project network from which the scheduling will flow (Figure 3-12). 

While there are other methods such as the Project Evaluation and Review 
Techniques (PERT) and the Critical Path Method (CPM) for developing a schedule, the 
authors promote their own process called "cards-on-the-wall" (COW) [Ref. 22: p. 163]. 
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System Components 

held together by 
“glue” (integration) 


Comp 

onent A 

Compo¬ 

nent 

B 

Compo¬ 

nent 

C 

Component 

D 


The whole does 
more than the 
sum of its parts 


Product 

Breakdown Structure 

Shows the components 
from which the 
system was formed 

j 


Work Breakdown 
Structure (WBS) 

All work components 
necessary to produce 
a complete system 



■■■■ 

Design Fab Assemble Test! 


■■■■ 

Training Test Data 

i " 1 i 

System Project 
Engineering Management 


Figure 3-11 From [Ref. 22: p. 159]. The Work Breakdown Structure. 
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Figure 3-12 From [Ref. 22: p. 162]. The Work Breakdown Structure Tasks Related to 

the Project Network and Schedule. 


The drawback to PERT and CPM is that they allow a single person, using automated 
tools, to complete the construction of the schedule. However, the COW method facilitates 
the interaction of the team because the members post the separate work packages, by 
project phase, on the wall for all to see. This method allows all of the members of the 
. team to congregate and provide input to the planning process as well as exposing them to 
the interdependencies of each work package. 

Using the COW method, the authors present several steps from which the 
scheduling process iterates [Ref. 22: p. 162]: 

-Combine the tasks to form a project network. 

-Define and evaluate risks. 

-Develop risk mitigation actions and add to the network. 

-Factor in task duration time. 

-Determine the critical path. 
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-Shorten the critical path. 

-Commit to performing to the task schedules. 

Devising a solid, coherent plan provides many benefits to the project 
manager. Interfacing with the other stakeholders, obtaining resource commitments, and 
reacting to unforeseen circumstances becomes easier because of the lack of ambiguity. In 
addition to the techniques mentioned in this section, the authors provide [Ref. 22: p. 153] 
a table of major elements and techniques for planning (Figure3-13). 


Key Element 
Products 


Tasks 

Strategy 


Network 


Schedules 


Resources 


Process 


Primary Technique 


Decomposing the deliverables into their hierarchi- Project Product List 
cal structure—all the way from the project require- and Fact Sheets 
ments down to the lowest level internal and 
external deliverables. 


Defining the tasks needed to complete each deliv- Work Breakdown 
erable. Structure 

Identifying the risks and opportunities and the Lessons Learned 

customer-compatible risk and opportunity strategy 
with preventive, causative, and contingent action 
plans. 


Logically arranging the required tasks to portray 
the best delivery approach. 


Cards-on-the-Wall net¬ 
work, followed by a 
computerized network 
and critical path deter¬ 
mination. 


Scheduling each task then refining and shortening Scheduling software 

the project's overall critical path through iterative 

steps. 

Establishing resources (personnel, equipment. Spread sheets and cost 
finances) needed to accomplish each task on estimating models 

schedule. 


Commitments Committing the necessary resources and funds for Project Work Authoriz- 
each task as determined from the schedule and task ing Agreements 
definition. 


Figure 3-13 From [Ref. 22: p. 153]. Major Elements and Techniques for Planning. 
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e) Risks and Opportunities 

Risk and opportunity management is a critical part of the planning process 
that is performed simultaneous with the project plan, however, it uses a separate and 
unique process. The authors provide specific steps for risk and opportunity management 
to allow a project manager to make conscious, distinct arguments for management 
actions. This structured process is ongoing and applied to the project as it evolves [Ref. 
22: p. 176]: 


-Identify the risks and opportunities. 

-Assess both probability and impact and then forecast the outcome. 

-Compare the outcomes and prioritize. 

-Develop feasible management actions to enhance opportunities 
and mitigate risks. 

-Estimate the cost of proposed immediate and contingent actions. 

-Compare resultant changes to weighted value against action costs. 

-Decide on actions required, and obtain concurrence. 

-Document and incorporate decisions in all planning. 

There are several methods available to help identify risks and 
opportunities. They range from off-core studies to scenario planning and failure mode 
analysis. While the techniques vary, the common principle is the same: to properly 
identify the risks and ppportunities, the project team must systematically apply an 
appropriate method through each phase of the cycle. [Ref. 22: p. 174] 

The Weighted Value (WV) method provides a good tool for the project 
manager to assess the probability and the impact of risk and opportunity. The WV is 
relatively simple to quantify. It is the probability of occurrence multiplied by the cost of 
the impact of that event occurring. For example, if the probability of an event occurring is 
.8, and the impact of that occurrence is $100,000, then the weighted value for that risk is 
$80,000. [Ref. 22: p. 182] 


43 



Quantifiable measures permit a project manager to continue with the risk 
and opportunity management process. The identified and quantified WV calculations 
allow the project manager to compare and prioritize events. Armed with a prioritized list 
of risk and opportunity events, management can then make a plan for influencing either 
the probability of occurrence or the impact on the outcome through the application of 
mitigating or enhancing activities. Once this process is complete, the mitigating actions 
can be incorporated into the project plan. 

f) Project Control 

Project control is process control. Further, process controls are needed at 
every level and in each project activity because projects without adequate process control 
usually fail. The authors present a dual system of process control designed for reducing 
risk [Ref. 22: p. 191]: 

-Baseline Control: Proactive control of the Project Plan and 
changes to the plan to help ensure that events happened as 
planned, and that events not planned do not happen. 

-Performance Control: Reactive control of variances in the 
performance of the Project Plan or corrective action taken when 
unplanned events do happen. 

For maximum effectiveness, controls need to be tailored to the project and 

in place before it begins. Proactive and reactive measures should be relevant, efficient, 

simple, and timely. To achieve these goals, the authors list five essential considerations to 

a control process [Ref. 22: p. 191]: 

-Things to be controlled: The function that must be controlled to a 
standard of performance. 

-Control Standard: The acceptable standard of performance. 

-Control Authority: The person or organization that imposes the 
standard and can grant exceptions. 
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-Control Mechanisms: The forum or device that measures and 
controls conformance to the standard. 

-Variance Detection: The identification of deviations of the control 
process or violations of the standard. 

An example of process control is the application of these considerations to 
"schedule control." In this example the schedule is the thing to be controlled, the master 
schedule is the standard, the business manager is the authority, the change board is the 
mechanism, and the status review provides the variance detection. [Ref. 22: p. 43] 

g) Project Visibility 

Project visibility is the means by which the project manager and project 
team know the status of their project activities. Project visibility techniques change as the 
project advances through the project cycle. They are determined by the timing, critical 
need, and geographic location of the required data. There are several techniques presented 
by the authors to gain visibility for project activities. Among these methods are meetings, 
reports, tiger teams, and glance management. [Ref. 22: p. 212] 

Glance management includes any number of informal follow-up 
techniques such as "management-by-walking-around" that provides awareness of a 
project's status with a quick glance. Other examples of this type include sampling work in 
progress, sitting in on lower level meetings and engaging in conversations before or after 
meetings. [Ref. 22: p. 213] 

Tiger teams are used to focus on trouble areas. Members of these teams 
are usually technical experts or experienced troubleshooters that can provide an objective, 
third party perspective of the problem. The ultimate objective of a tiger team is to identify 
the cause of a problem without trying to assess the blame. [Ref. 22: p. 216] 
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h) Project Status 

Project statusing is the measurement of the progress of the project against 
the project plan. The goal is to identify variances that require corrective action to restore 
the plan. The authors contend that an effective process for statusing [Ref. 22: p. 224]: 

-Collects essential information only. 

-Measures primary variables that impact results. 

-Tailors information to the needs of the team members interpreting 
it. 

Project managers should always know the status of four primary factors: 
schedule, technical, cost, and business. Table 3-2 provides the key metrics for each of 
these factors. [Ref. 22: p. 225] 


Schedule 

Technical 

Cost 

Business 

Progress summary 

Development results 

Actual versus budget 

Contract change process 

Master schedule 

Design release 

Headcount 

Actions to/from customers 

Milestone 

accomplishments 

Technical review 

Earned value vs. 
expenditures 

Action to/from management 

Assemblies and 
modules 

Technical performance 
measurements 

Bum rate and overtime 
ratio 

Actions to/from contractors 

Tasks 

Interface control 

Estimate to completion 

Funding 

Subcontractors 

Quality 

Estimate at completion 

Top-ten problems 

Parts and material 


Profit 

Security clearances 

Earned value 


Dispersion ratio 

Project manager's assessment 


Table 3-2. Status Factors and their Key Elements. 


The authors provide several examples of methods to measure the status of 
these key factors. One of these methods is to use the quantity of milestones as a schedule 
metric (Figure 3-14) [Ref. 22: p. 228]. Another example provided is a Total and 
Experienced Headcount variance report to anticipate cost and schedule problems (Figure 
3-15) [Ref. 22: p. 229]. 
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Number of Milestones 


■■ Planned 
■ ■ Actual 


Figure 3-14 From (Ref. 22: p. 231]. Headcount Variance Report. 



Figure 3-15 From [Ref. 22: p. 228]. Milestone Status Report. 
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One of the more prevalent systems used today to refine performance 
measurements is the Earned Value (EV) method. While the EV method requires detailed 
pl annin g to establish and maintain, it provides a manager with a powerful tool to provide 
early warning of problems by relating cost, schedule and technical progress. Figure 3-16 
illustrates an application of the EV method to a project. [Ref. 22: p. 233] 

In this example, the metrics denotes that the project depicted in Figure 3- 
16 is behind schedule and over budget. The variance between the Budgeted Cost of Work 
Performed (BCWP) and the Actual Cost of Work Performed (ACWP) signifies that the 
actual costs of completing the tasks exceeded the budgeted costs and therefore the project 
has expended more funds than anticipated at that point in the schedule. The variance 
between the BCWP and the Budgeted Cost of Work Scheduled (BCWS) connotes that the 
project has performed less work than scheduled. Through comparisons of these and other 
metrics, the project manager can interpret trends and anticipate the eventual cost of the 
completed project. 



Figure 3-16 From [Ref. 22: p. 233]. Earned Value Chart. 



i) Corrective A ction 

Corrective actions are those steps management takes to restore the project 
to the plan. These actions are usually triggered by unacceptable variances discovered 
through the use of statusing techniques. By monitoring variance and recognizing an 
imminent or actual breach of a predetermined threshold, a manager maintains the 
possibility of regaining lost control over a project. By taking corrective action in a timely 
manner, a manager can restore control of the project. 

According to the authors, the steps for corrective are as follows [Ref. 22: 


P- 241]: 


-Analyze the problem, its current impact, and the growth of the 
impact if no action is taken. 

-Prioritize project problems from the most to the least serious. 

-Determine the best approach using analytical decision analysis. 

Examples of corrective actions are overtime, added work shifts, or an 

alternate technical approach. Figure 3-17 illustrates the use of an analytical tool to help 

evaluate alternatives for corrective action that requires adjusting the work shifts. 


Evaluation 

Criteria 


Alternative 1 
One 12 hr shift 


Alternative 2 
Two 8 hr shifts 


Alternative 3 
iThree 8 hr shifts! 


Alternative 4 
Two 12 hr shifts! 


Musts (Go-No Go): 

Certified Software Testers 

Available within 3 weeks 


Wants 


Weight 

(W) 


Comments 


Score 

S ' 5 
<r 


j Comments 


Score 

i ' 5 

cc 


Comments 


Score 

i' 5 

cc 


Comments 


Score 

« 
cc 


Factors 

Maximizes 

productivity 

Highly experienced 
in our software 

Low average labor 
rate 


10 

8 

8 


10] 80 


| 100 | 

64 

40 


Max Score (lOxW) 

Total Score _ 


260 


186 


214 


204 ] 


Figure 3-17 From [Ref. 22: p. 244]. Evaluation by Weighted Scoring. 
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j) Project Leadership 

According to the authors [Ref. 22: p. 44], Project Leadership is the most 
important of the 10 project management elements. Project Leadership is the bonding 
agent that holds the previous nine project management elements together. It is through the 
proper exercise of leadership that the other nine elements are used. Three aspects of 
project leadership are addressed [Ref. 22: p. 246]: 

-Situational leadership model, or the relationship of leadership to 
management. 

-Techniques for inspiring and motivating individual and group 
performance. 

-Style, or determining and communicating your leadership style. 

To portray the situational nature of leadership, the authors use an 
orthogonal model (Figure 3-18). The cylinder represents the sequential actions for a 
typical task. The disk represents the project management elements held together by the 
rim of project leadership. The orthogonal positioning of the techniques of leadership 
represent their situational application to the project relative to the project cycle and 
current circumstances. 

To motivate the project team, project managers should apply several 
techniques. They need to communicate their vision for success to the project team. They 
must also create the environment to enable the team member to achieve the stated vision 
or goals and recognize team members for their good performance. Project managers 
especially need to draw on their subordinates' strengths as well minimizing their 
weaknesses. Finally, project managers need to set the example by acting as they want the 
team to act. 
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Figure 3-18 From [Ref. 22: p. 250]. Orthogonal Project Management Model. 

D. CHAPTER SUMMARY 

The program management model is critical to the DoD Acquisition system 
because it provides a single point of contact that acts as an "integrator" for a complex 
system of related functional disciplines. 

The criticality and complexity of program management makes it an ideal topic for 
which to develop an automated program that incorporates a qualitative review capability 
currently lacking within the DAU course offerings. The remainder of this thesis focuses 
on developing such a program, which will be referred to as an "Automated Tool for 
Acquisition Program Management Students" (ATAPMS). 

The next chapter details the content, structure, and topic requirements for an 
ATAPMS model tailored to the needs of instructors at the NPS. 
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IV. STRUCTURE AND CONTENT REQUIREMENTS 

A. INTRODUCTION 

This chapter develops the structure, strategies and topics for an ATAPMS tailored 
to the NPS. Because the Vee-model described in chapter three is a useful concept, this 
thesis uses its requirements definition principles as a guide to help determine the 
requirements and specifications for an ATAPMS model. 

B. STRUCTURE 

The structure defines the top-level requirements for the application of an 
ATAPMS tailored to the NPS. The first step for determining the structure corresponds to 
the first step in the Vee-model: understand the user’s requirements [Ref. 22: p. 36]. 
Feeding into the user requirements and system concept phase of the Vee-model is the 
“Statement of User’s Needs” [Ref. 22: p. 187]. In the case of this application of an 
ATAPMS, the NPS requires an automated program to use in a laboratory setting to assist 
with the instruction of Acquisition Program Management. The users for this ATAPMS 
are identified as the instructors in the Systems Management Department of the NPS who 
teach Acquisition Program Management related courses. Personal interviews conducted 
with six of these faculty members verified several of the important characteristics 
necessary to determining the overall structure of an NPS ATAPMS. 

While there were several areas of concern relating to the overall structure, based 
on the interviews, the most relevant to an ATAPMS are the approach, setting, scope, 
scalability, and distance learning considerations. 
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1. Approach 

There are two basic approaches evident in the automated tools used by the DAU 
consortium schools to provide instruction to students: tutorial and capstone. 

The tutorial approach is designed to expose the student to new information. This 
is the approach currently used by the DSMC in introductory courses like Acquisition 101 
that are accessed through the Internet. [Ref. 23: p. 1] 

The other method is the capstone approach exemplified by the SAFE application. 
This method does not try to teach the student new material, instead it reinforces or 
integrates concepts that the students learned as a prerequisite to participating in the class. 
The capstone approach reinforces these concepts by requiring the students to apply 
learned concepts to the problems presented by a real world scenario. 

The NPS ATAPMS should follow the capstone approach because according to the 
instructors interviewed, there is a greater need for that type of application than for the 
tutorial method. In fact, other than the SAFE program, there are no capstone type 
automated programs used at the NPS or within the DAU consortium to assist with the 
instruction of Acquisition Program Management. 

2. Scope 

Because the NPS instructors will be the primary users, the ATAPMS should 
accommodate the NPS quarterly course class length. Courses are taught at the NPS on a 
quarterly schedule with each quarter 12 weeks in length. Generally, the first and last 
weeks are not conducive to conducting laboratory exercises, so the scope of the 
ATAPMS should be to accommodate 10 weekly sessions. 
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3. Setting 

There are three different scenarios likely for students using an ATAPMS. Two of 
these scenarios involve the resident classroom laboratory environment and one involves 
an ATAPMS used in a non-resident distance learning capacity. 

a) Resident Students 

One of the classroom scenarios involves accessing the module for the first 
time beginning at the start of that week’s laboratory session, and the other involves 
students accessing the module’s material a day or more prior to the start of the lab 
session. The NPS ATAPMS should be designed to incorporate the latter setting. The 
advantage to allowing students to access materials before the requirements are due is that 
the students will have more time and opportunity to work with the material. This 
additional time will also allow for the inclusion of more detailed scenarios and should 
ultimately translate to a more meaningful learning experience. 

b) Non-Resident Students 

The second scenario envisions the use of the ATAPMS to support distance 
learning. A program designed with more involved scenarios and thought provoking 
material for resident students will accommodate non-resident students as well. Designing 
the ATAPMS to include more detailed material allows for an easier migration from the 
traditional resident lab sessions to distance learning sessions. This is in part because 
detailed course material will naturally support the non-resident students enabling them to 
work more autonomously. 
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4. Scalability 

One theme evident from the research is that there does not appear to be any one 
specific design solution to fit the acquisition academic communities’ requirements. Of the 
six instructors interviewed, each had subject matter topics and strategies for presenting 
those topics that were unique to their needs. They also had students with a variety of 
educational backgrounds who were pursuing different tracks in the acquisition field. 

To be useful, an ATAPMS must be flexible enough to conform to each 
instructor’s needs. One method of accomplishing this flexibility is to design the 
ATAPMS using a modular architecture that allows the replacement or addition of any one 
module without affecting the rest of the program. With this design format, each week’s 
session would be a stand-alone module providing the instructors a “plug and play” 
capability. Given this flexible format, instructors would have the ability to select the 
appropriate modules and present them in the most appropriate order to best tailor their 
learning objectives to their category of students. [Ref. 24] 

Another advantage to the modular structure is the ability to add new modules on 
an as needed base with minimal cost or disruption to the existing program [Ref. 24], This 
feature will allow an ATAPMS to stay relevant and to continue to serve the academic 
classroom by adjusting to changes in the acquisition field. In contrast, the SAFE program 
discussed in a previous chapter does not share these advantages, and its inflexibility is a 
common complaint shared by a number of the instructors interviewed. 
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5. Distance Learning Considerations 

Even though the current users for the ATAPMS are resident students, the intent of 
EO 13111 and the Defense Reform Initiative are clear: DoD schools need to take 
advantage of technology to help make education more accessible and at less cost. 

Several of the schools within the DAU consortium already use distance learning 
technology to some extent. NPS has provided video teleconferencing courses for several 
years [Ref. 25]. The Defense Systems Management College [Ref. 23] and the Information 
Resources Management College [Ref. 26] are just two of the DAU consortium members 
that provide computer based distance learning courses. To insure that the ATAPMS 
meets Presidential and DoD intentions and that it is capable of supporting the distance 
learning requirements of the future, it should be compliant with Instructional 
Management Systems standards and be designed to meet the end-state goals of the 
Advanced Distance Learning Initiative. 

The vision of the ADL initiative is to “provide access to the highest quality 
education and training, tailored to the individual’s needs, anywhere and anytime in the 
world.”[Ref. 27] The ADL initiative attempts to achieve this vision by transitioning from 
the current model of training which is classroom centered to a model that is network 
centered. To foster the network centered model, the ADL initiative established several 
goals that a distance learning application should meet. These end-state goals are: 
accessibility, interoperability, durability, re-usability, and cost effectiveness. [Ref. 28] 
a) Accessibility 

Accessibility refers to the student’s ability to have access to the course 
modules from a variety of remote locations. Ensuring modules are designed with the 
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capability to be accessible through the Internet is the current method of meeting this goal 
[Ref. 15]. 

b) Interoperability 

Interoperability refers to the student’s ability to use a module at his 
location even though the module was developed at a different location on a different 
platform or with a different set of tools. Current indications from the software industry 
are that an object-based approach will provide the basis for this platform neutral goal 
[Ref. 16]. 

c) Durability 

A durable module is a module that is robust. It requires no re-work or re¬ 
coding when the base technology changes. 

d) Re-usability 

Re-usability refers to the capability of the module to be used in multiple 
applications. The benefit will be a module that saves development time and money. The 
same object oriented approach necessary for interoperability also provides the basis for 
re-usability [Ref. 13]. 

6. Baseline Summary 

Using the Vee-model method, it is now possible to establish a baseline for the 
structure of the NPS ATAPMS before determining the content. The following list 
summarizes the proposed requirements for the ATAPMS’ top-level structure: 

-Use a capstone style approach to the content. 

-Accommodate at least 10 weekly topic modules. 

-Capable of presenting well developed thought-provoking scenarios and 
questions. 

-Ensure modular architecture for scalability and a plug and play capability. 
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-IMS compliant. 

-Ensure Accessibility. 

-Ensure Durability. 

-Ensure Re-usability. 

-Ensure Cost effectiveness. 

C. CONTENT 

Having determined the structural requirements, it is now possible to develop a set 
of top-level content requirements. Because the research revealed that the content 
requirements were so varied, and the proposed structure is flexible enough to 
accommodate them all, there is no need at this time to propose a final solution set for the 
content requirements. Therefore, it is not within the scope of this thesis to attempt to 
mature the content requirements to the point of recommending them as the approved 
baseline. The content proposed in this section will address the most recurring content 
needs as revealed by the research, and subsequent sections of this thesis will draw from 
this content for the purpose of off-core prototyping as a proof of concept measure. 

Through the course of the research, it became evident that the content consisted of 
two distinct parts: topics and strategies for presenting the topics. An analysis of the 
literature review and instructor interviews revealed that material for instructing 
Acquisition Program Management can easily be grouped or categorized by logical topics. 
However, depending on the objectives for a particular course, the material included or 
emphasized within the topic presented to the student may be different. For example, 
Acquisition Strategy is a topic that equates to the Project Cycle described in chapter three. 
The material related to timeline and deliverable expectations used in this topic will differ 
if the objective is to emphasize a Major Automation Information System instead of a 
Major Defense Acquisition Program [Ref. 29]. The following sections present a listing of 
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topics and strategies that are not meant to be all-inclusive, but they are representative of 
the recurring information derived from the literature review and instructor interviews. 

1. Topics 

The explanation of each topic presented in this section will present a higher level, 

more generalized description because each topic’s content is influenced by the 

relationship between the topic and the presentation strategy. The intent of this section is 

to refine the major topics within Acquisition Program Management only to the level 

necessary to develop a proof of concept prototype program in the subsequent sections. 

This development strategy is consistent with the off-core activity method presented in the 

Vee-model concept that states that “off-core activities do not seek a final solution, but 

rather a demonstration that one is feasible.” [Ref. 22: p. 188] 

When reviewing the list of candidate topics, the lack of topics related to 

Acquisition Reform themes is conspicuous. They are not listed as individual topics 

because, in the words of former Secretary of Defense William Perry: 

These policies are fundamental to all of our acquisition activities essential 
to clearly defining our performance requirements, establishing affordable 
life-cycle cost requirements, and working together to develop affordable 
and executable strategies to resolve issues in a timely manner. [Ref. 30] 

Using this principle as a guide, there is no need to include the reform themes as 

separate topics, instead each of the following topics should incorporate the reform themes 

into its content. 
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a) Acquisition Strategy for a Major Defense Acquisition Program 

(MDAP) 

The Acquisition Strategy is related to the Project Cycle described in 
chapter three and is defined as [Ref. 31: p. 1-1]: 

...a business and technical management approach designed to 
achieve program objectives within specified resource constraints. It is the 
framework for planning, organizing, staffing, coordinating, and leading a 
program. It provides a master schedule for research, development, test, 
production, fielding, and other activities essential for program success, and 
for formulating functional strategies and plans. 

This topic will use a Major Defense Acquisition Program for the focus of 


its content. 


b) Acquisition Strategy for a Major Automation Information 
System (MAIS) 

An Acquisition Strategy for an MAIS contains the same conceptual bases 
for the overall plan for program execution as an MDAP. In fact, the DSMC’s Acquisition 
Strategy Guide [Ref. 31] makes no distinction between developing an Acquisition 
Strategy for an MDAP or for an MAIS. However, there is enough of a distinction between 
the two types of programs [Ref. 29] that the Acquisition Strategy for an MAIS does 
warrant a different emphasis and therefore its own topic. 

c) System Engineering Process 

The System Engineering Process is used to [Ref. 32: 4.3]: 

.. .transform operational needs and requirements into an integrated 
system design solution through concurrent consideration of all life-cycle 
needs (i.e., development, manufacturing, test and evaluation, verification, 
deployment, operations, support, training and disposal). The system 
Engineering process shall establish a proper balance between performance, 
risk, cost, and schedule, employing a top-down iterative process of 
requirements analysis, functional analysis and allocation, design synthesis 
and verification, and system analysis and control. 
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The System Engineering Process is closely related to the Technical Aspect 
of the Project Cycle. In fact, Forsberg, Mooz, and Cotterman define the System 
Engineering Process as the application of the two subprocesses within their Vee-model: 
the System Analysis and Design Process and the System Integration and Verification 
Process. [Ref. 22: p. 37] 

d) Defining Critical Design & Capabilities 

This topic encompasses the objectives of the original SAFE program and 
could easily be developed into three or four sequential modules. The overall intent is to 
allow students to exercise their analytic abilities and knowledge of systems acquisition. 
The module(s) should allow students to make trade-off decisions concerning the cost, 
schedule and performance aspects at key decision points. The listing of specific tasks 
include [Ref. 33]: 

-Conduct requirements and trade-off analysis. 

-Identify, analyze, and assess program risks. 

-Evaluate contractor proposals. 

-Develop cost estimates, budget submissions, acquisition 

strategies, and program baselines. 

-Identify critical acquisition management issues and prepare 

recommendations for decision-makers. 

This topic is a compilation of several of the Project Management Elements 
and the Project Cycle's Budgeting, Business, and Technical aspects. 

e) Risk Management 

DoD 5000.2-R states that Program Managers need to institute a risk 
management program for each acquisition program “to identify and control performance, 
cost, and schedule risks.” [Ref. 32: 3.3.2] Risk Management in the DoD parallels the 
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Risks and Opportunities element of the Project Management Elements and is defined as 
“All plans and actions taken to identify, assess, mitigate, and continuously track, control, 
and document program risks.” [Ref. 3] 

j) Analysis & Control 

While Risk Management is the process to identify, assess, track, and 
mitigate critical items, Analysis and Control activities serve as a basis for measuring 
progress [Ref. 34: p. 5-3]: 

DoD 5000.2R, Part 4 states that performance metrics must be 
developed and maintained throughout the process (SE process). These 
metrics will be used to assess how well the evolving design meets the 
system requirements...Particular attention will be paid to those metrics that 
are critical to risk management...The data for each metric will be based on 
engineering judgment, design analysis, test data (including early test 
results), and operational data, depending on the status of the design. 

Analysis and Control is related to three of the Project Management 
Elements: Project Control, Project Visibility, and Project Status. 

g) Acquisition Logistics 

Acquisition Logistics is a topic so broad that it could easily provide 
material for several modules. It is broad because it is a multi-functional technical 
management discipline that encompasses design, development, test, production, fielding, 
sustainment, and improvement modifications that achieve the user’s peacetime and 
wartime readiness requirements [Ref. 35: 4-1]. DoD Regulation 5000.2-R mandates that 
[Ref. 32: 4.3.3]: 

The PM shall conduct acquisition logistics management activities 
throughout the system development to ensure the design and acquisition of 
systems that can be cost-effectively supported and to ensure that these 
systems are provided to the user with the necessary support infrastructure 
for achieving the user's peacetime and wartime readiness requirements. 
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Because Acquisition Logistics is so broad in scope, it can be related to 
several of the concepts and elements in the Project cycle and the Project Management 
Elements as it is applied throughout system development as mandated by the DoD. 

h) Contracting & Contract Management 

There are a myriad of contract types. The specific type is selected to 
provide for a reasonable sharing of risk for the performance of the contract. Because of its 
ability to balance the risk, the selection of the most appropriate type of contract and fee 
for the particular procurement is essential. [Ref. 36: p. 1-1] 

This topic corresponds to the Source Selection Phase of the 
Implementation Period from the Project Cycle. 

i) Budgeting 

Budgeting involves the short and long range planning and execution of 
resources. The improper application of this topic means that there may be no program 
[Ref. 32: 2.5.2]: 

No acquisition program shall be approved to proceed beyond 
program initiation unless sufficient resources, including manpower, are 
programmed in the most recently approved FYDP, or will be programmed 
in the next Program Objective Memorandum (POM), Budget Estimate 
Submission (BES), or President's Budget (PB). 

Budgeting in acquisition relates to the budgeting aspect of the Project 


Cycle. 
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j) Test & Evaluation 

Test and evaluation (T&E) in a defense system’s development and 
acquisition program helps to identify the areas of risk to be reduced or eliminated [Ref. 
37: p. 1-1]: 

Test and evaluation programs shall be structured to provide 
essential information to decision-makers, verify attainment of technical 
performance parameters, and to determine whether systems are 
operationally effective, suitable, and survivable for intended use. 

This topic corresponds to the Verification Phase of the Implementation 
Period from the Project Cycle. 

k) Production & Fielding 

During this phase of the DoD acquisition process, the Program Manager 
must ensure the system meets the user’s needs while being sensitive to the impact of the 
manufacturing decisions on production costs. Through careful selection from the 
manufacturing alternatives, the PM can achieve significant cost savings. The key 
activities for this phase include manufacturing, contract monitoring, and acceptance 
testing. [Ref. 38: p. 3-11] 

This topic corresponds to the Production Phase of the Project Cycle. 

l) Work Breakdown Structure & Earned Value 

The DoD regulation 5000.2-R requires Program Managers to establish a 
program work breakdown structure (WBS) to provide a framework for program and 
technical planning, cost estimating, resource allocations, performance measurements, and 
status reporting [Ref. 32: 4.4.2]. 
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The WBS corresponds to the Project Planning element and EV 
corresponds to the Project Status element of the Project Management Elements. 

m) Software Management 

Software Managers need to be able to “define objectives, to create, 
evaluate, and select alternatives for reaching them, and to control the implementation of 
selected alternatives”. [Ref. 39] These can be difficult tasks because of the intangible 
nature of software. Software estimation is a key activity in planning and managing 
software intensive programs. Software cost and schedule estimation is usually 
implemented using models that quantitatively relate measures or projections of software 
size, development and maintenance effort, schedule, and quality through the use of an 
integrated performance or efficiency measure. [Ref. 40] 

Software Management is not limited to any single portion of the project 
management principles from chapter three; instead, one would use all the principles while 
emphasizing the software relevant elements like Project Control, to effectively manage a 
software project. 

n) People Factors 

This item was not offered in any of the preexisting courses. However, 
through the research and interviews it surfaced as a necessary topic. People skills are 
required for a manager when building or interacting with the project team and for 
communicating with the stakeholders for a project. This topic is related to the Teamwork 
and Leadership elements. 
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2. Strategies 

Because of the varied student backgrounds, class objectives, and instructor 
preferences, there emerged no single strategy appropriate for presenting the modules. 
Fortunately, a flexible, modular architecture as described in a previous section will allow 
an instructor to use whatever strategy he or she determines is the most appropriate to 
reach the course obj ectives. 

The research indicated that there were several strategies to presenting the topics 
that seemed to be viable options for meeting course objectives. 

a) Chronological by Phase 

Chronological by phase is a common approach to teaching Program 
Management. Most of the consortium schools teaching Acquisition Program Management 
used this method in at least one of their courses. This approach takes the student through 
the acquisition phases in a sequential manner, introducing each topic as it appears in 
chronological order. For example, the first topic may be a Requirements Generation 
related topic followed by a Concept Exploration related topic. 

b) Concept Centric 

A concept centric strategy places the importance of the topic or concept 
over the need for a chronological ordering of their appearance. Using this strategy, the 
topics are more likely to appear as stand alone material and not lead into the next module. 
For example, a topic concerning logistics may follow a topic involving test and 
evaluation. 
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c) Parallel Comparison of MDAP vs. MAIS 

The comparison strategy emphasizes both types of program through out 
the acquisition cycle to provide an appreciation for their differences. Modules designed 
for this strategy would probably be sequential in nature and the objectives would focus on 
highlighting the differences between the management of these programs. 

d) Systems Engineering Centric 

A systems engineering centric approach provides students a view of the 
acquisition process through modules that underscore the role of the systems engineering 
process. 

e) Logistics Centric 

A logistics centric approach uses the same concept as the systems 
engineering centric strategy except it features the acquisition logistic perspective to the 
acquisition systems management process. 

f) Hybrid Approach 

A hybrid approach mixes elements from the preceding strategies. This 
style may involve any number of the aforementioned approaches combined to present an 
optimized collection of strategies for one course. An example of this strategy would be to 
use a parallel comparison for the first two modules, a chronological by phase for the next 
few modules, and then a few concept centric for the final modules. 

D. CHAPTER SUMMARY 

Because it is impractical for a single static program to address all the strategies 
and topics required by the users, the NPS ATAPMS needs to accommodate many 
interchangeable modules and must scale to the user’s needs. Additionally, to be viable for 
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the future and to comply with the intent of the Defense Reform Initiative, the program 
must also incorporate the Advanced Distance Learning precepts. Table 4-1 provides a 
comprehensive summary of the recommendations for the structure, strategies, and topics 
for the application of an ATAPMS at the NPS presented in this chapter. 

This chapter used the Vee-model concept to develop baseline recommendations 
for the structure, and to propose content for off-core study. The next chapter will continue 
this method by using a subset of the proposed content to develop a prototype of an 
ATAPMS for the NPS. 


Structure 

Content 


-Use a capstone style approach 
to the content 
-Accommodate at least 10 
weekly topic modules 
-Be capable of presenting well 
developed thought- 
provoking scenarios and 
questions 

-Ensure modular architecture 
for scalability and a plug and 
play capability 
-Be IMS compliant 
-Ensure Accessibility 
-Ensure Durability 
-Ensure Re-usability 
-Ensure Cost effectiveness 

-Acquisition Strategy for a Major Defense 
Acquisition Program (MDAP) 

-Acquisition Strategy for a Major 

Automation Information System (MAIS) 
-System Engineering Process 
-Defining Critical Design & Capabilities 
-Analysis & Control 
-Risk Management 
-Acquisition Logistics 
-Contracting & Contract Management 
-Budgeting 
-Test & Evaluation 
-Production & Fielding 
-Work Breakdown Structure & Earned Value 
-Software Management 
-People Factors 

-Chronological by Phase 
-Concept Centric 
-Systems Engineering 
Centric 

-Logistics Centric 
-Hybrid Approach 


Table 4-1. Summary Table. 


69 








THIS PAGE INTENTIONALLY LEFT BLANK 


70 



V. DESIGN OF THE PROGRAM MANAGEMENT TRAINING TOOL 


A. INTRODUCTION 

This chapter continues the Vee-model off-core study concept by exploring one 
possible approach to devising an ATAPMS application that meets the requirements 
specified in Chapter Four. Specifically, this chapter enumerates strategies for acquiring 
the software required to support an ATAPMS and then uses one of those strategies for 
demonstrating the construction of a notional module. 

B. DEVELOPMENT STRATEGIES 

There are two realistic approaches to developing an ATAPMS program that meet 
the top-level specifications defined in the previous chapter: develop new software code or 
use commercial based authoring tools. 

A third option, modifying existing software by combining applications such as 
Microsoft Project Manager and Microsoft Access using a web based interface like Cold 
Fusion, is not reasonably feasible. Adding additional modules in a "plug and play" 
fashion that incorporates the IMS requirements to applications cobbled together is as 
involved as writing new software code. Further, software licensing management, 
seemingly random changing of application versions, and lack of vendor support because 
of modifications made to their products, increases the undesirability of this option. 

The decision between the two remaining options for the best approach depends on 
several factors. In the case of the NPS, the most applicable factors according to the 
interviews were: 

-Ease of use: The ability for an instructor to use the software to create or 
modify modules. 
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-Applicability: The capability of the program to help the instructor meet 
his or her learning objectives. 

-Cost: The amount of resources measured in time and money required to 
create modules. 

-Maintainability: The ability to modify, correct faults, improve 
performance, or other attributes, or adapt to a changed environment. 

-Time: The amount of time required to field the initial and subsequent 
modules. 

1. Developing New Code 

The first method is developing the application using completely new software 
code. This method has one strong benefit. An application designed from the beginning 
with the specifications and requirements of the NPS would optimize the “applicability” 
criterion, and as a result be highly tailored to the needs of the acquisition program 
management instructors. 

According to an NPS software instructor interviewed, however, the disadvantages 
to building an application to meet the top-level specifications identified for an ATAPMS 
by writing new code are numerous. Time required for development of the initial and 
subsequent modules is a significant factor for this option. Also, to obtain the ease of use 
and applicability desired, the cost for building a proprietary program will be much higher 
than purchasing a commercial product. 

Maintenance requirements add another heavy cost burden to the builder as well. 
For example, if the ATAPMS were programmed in JAVA, a language that adheres to the 
top-level specifications enumerated in the previous chapter, all content authors needing to 
create or modify existing or subsequent modules must adhere to the same standards used 
to create the original program. While still feasible, this restriction makes the concept of a 
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“plug and play” program harder to accomplish, especially as institutional knowledge of 
the standards for the original program declines over time. 

Additionally, the ability for the average instructor to create and employ a module 
becomes highly unlikely in this scenario. Because of the requirement to write code 
specific to the original program, modifications to existing modules or the creation of 
additional modules will probably require the efforts of an experienced programmer. 

Finally, the adherence of the software to the IMS standards adds a complex and 
costly dimension to the generation of the software code. Not only will the modules 
require a programmer’s skill to write, but that programmer must also be familiar with the 
nuances of the IMS standards. 

2. Commercial Based Software 

The second option for developing the ATAPMS application is to use 
commercially available authoring products. Authoring systems are software applications 
that allow the developer to create a multimedia education product without having to write 
software code. Each authoring system has its strengths: some work best with databases, 
computer-based instruction, or just as general all-purpose applications. [Ref. 41] 

The commercial authoring systems have one major disadvantage, but they also 
have several distinct advantages. The disadvantage to using a commercial product is the 
“applicability” factor. Because the functionality of a commercial product is 
predetermined, there may be features that an instructor needs that are not available. This 
disadvantage caused Indiana University to choose creating its online course management 
system by writing new code instead of adapting a commercially available application 
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[Ref. 42]. Although the Indiana University’s requirements are different from the NPS 
requirements, this example illustrates the importance of the “applicability” issue. 

The advantages to using commercial authoring tools that allow one to create 
custom courseware are numerous. The first advantages are cost and time. The average 
cost for a commercial authoring system typically range from $1,000.00 to $5,000.00 [Ref. 
41], which according to one NPS software instructor is several times less than writing 
new software code for a project like an ATAPMS. That cost differentiation widens 
considerably with the need for new modules and support for the existing software. Also, 
the time needed to develop the initial application or subsequent modules is measured in 
days or hours, not months. This advantage is especially significant for instructors who 
need to make last minute modifications to their courses. 

The third advantage is maintainability. The commercial vendors have an inherent 
need to keep pace with the evolving technologies in a cost effective and timely manner. 
This condition works to the user's favor because it shifts the burden for maintaining the 
software code from the user to the vendor. Because the vendor has a comparative 
advantage in maintaining the software, it will cost less for the user to adapt the existing 
software to a changed environment. 

A fourth advantage is the ability for the average instructor to generate his or her 
own modules without the need for programming expertise. The web based authoring tools 
that meet the ATAPMS top-level specifications use a graphical user interface that allow 
the user to create a multimedia product without having to write software code. 

A final advantage is the capability for the NPS to eventually integrate the use of 
an on-line management software system to organize the delivery of the ATAPMS 
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courseware. While this criterion was not mentioned by any of the instructors interviewed, 
the research indicates that given sufficient growth in the use of an ATAPMS, an on-line 
management tool may be essential to successfully scaling the ATAPMS to a wider 
audience. If the use of the ATAPMS grows from a once a week resident laboratory 
exercise to a distance learning course supplement, instructors may need such a tool to 
help administer the ATAPMS and other related courseware. The management system 
would assist in administering the ATAPMS by providing instructors the ability to 
structure learning groups, add users, control the course availability, and obtain detailed 
evaluation reports on student performance. [Ref. 43] 


The research demonstrates that to meet the initial needs of the NPS, investing in a 
commercial authoring tool to develop an ATAPMS has an advantage in many areas over 
designing a proprietary application (Table 5-1). 



Ease of 
Use 

Applicability 

Time 

Cost 

Maintainability 

Management 

System 

Commercial 

Authoring 

Tool 

4- 

- 

+ 

+ 

+ 

+ 

Proprietary 

Application 

+ 

+ 

— 

- 

— 

- 


Table 5-1. Option Comparison. 


C. CONSTRUCTING THE MODULES 

This section uses a subset of the proposed content from the previous chapter and a 
commercial authoring tool to illustrate one way of developing an ATAPMS. The 
following off-core study uses the “hybrid” approach, mixing the elements from the 
preceding types of strategies documented in Chapter Four to present the topic modules. In 
the planning phase, the titles of 10 modules are listed to illustrate a notional course load 
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for a quarter. From these 10 topics, four modules are presented in a high level format to 
demonstrate their learning points. In the layout phase, one of these modules is further 
developed using a commercial authoring application. 

1. Planning 


Table 5-2 depicts the 10 modules selected from the topic section of the previous 
chapter. 


Week 1 

Week 2 

Week 3 

Week 4 

Week 5 

Acquisition 

Strategy (MDAP) 

Acquisition 

Strategy (MAIS) 

System 

Engineering 

Process 

Define Critical 
Design & 
Capabilities I 

Define Critical 
Design & 
Capabilities II 

Week 6 

Week 7 

Week 8 

Week 9 

Week 10 

Define Critical 
Design & 
Capabilities III 

Risk Management 

Analysis & Control 

Acquisition 

Logistics 

Contracting & 

Contract 

Management 


Table 5-2. Notional Course Load. 


To continue the planning process, the learning objectives for modules 1,2,7 and 8 - 
are described using the U.S. Army’s “Task, Condition, Standards” method (Table 5-3), 
followed by a "design approach" (Table 5-4) description for constructing the modules. 
The task defines a measurable activity for the students to perform [Ref. 44]. The 
conditions explain the circumstances and environment in which the task is to be 
performed. The standard describes the minimum acceptable proficiency required in the 
performance of the particular training task. Finally, the design approach description 
provides general guidance for features and functionality required from the software to 
allow the module to accomplish the intended learning objectives. 
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Module 1 Module 2 Module 7 Module 8 

Acquisition Strategy Acquisition Risk Management Analysis and Control 

(MDAP) _ Strategy (MAIS) _ 

TASK: TASK: TASK: TASK: 

Develop an Develop an Evaluate a program Employ metrics 

Acquisition strategy for a Acquisition strategy for scenario and determine developed during Risk 

Major Defense a Major Acquisition the areas of risk. Management Module 

Acquisition Program. Information System. prioritize the selected and interpret results from 

areas and devise a risk a visual display of the 
mitigation strategy that submitted metrics. 

_ includes metrics. _ 

CONDITION: CONDITION: CONDITION: CONDITION: 

Given a Mission Needs Given a Mission Given a program Given a program 

Statement, an Needs Statement, an scenario with sufficient scenario that provides 

Acquisition Decision Acquisition Decision cost and probability of quantifiable data 

Memorandum, Progress Memorandum, Progress occurrence details, risk reflecting a snapshot in 

Report (CE Phase Report (CE Phase reduction strategies and time for a project status. 

Studies), Performance Studies), Performance a risk reduction budget. 

Requirements, and a Requirements, and a 

general scenario. _ general scenario. _ 

STANDARD: STANDARD: STANDARD: STANDARD: 

Student teams must Student teams must Student teams must Student teams must 

determine an appropriate determine an demonstrate a working analyze given program 

Acquisition strategy for appropriate Acquisition level of knowledge of data and extract the 

an MDAP. They must strategy for an MAIS. Risk Management. appropriate information 

adequately address the They must adequately Teams should be able to needed for input to a 

following elements in address the following analyze a scenario and mechanism that will 

their strategy: elements in their identify the areas of risk provide a visual read-out 

-basic strategy (in terms strategy: and produce a Risk of the submitted metrics, 

of milestones, phases, -basic strategy (in terms Management strategy Students must evaluate 

technical reviews, of milestones, phases, (to include a the visual output and 

contract award dates and technical reviews, quantifiable measure of provide mid-level 

other key events) contract award dates risk for each event). management analysis and 

-Streamlining, and other key events) Teams will perform a recommendations to 

-Competition -Streamlining, cost/benefit analysis. senior level management. 

-Contract types -Competition Teams will also identify 

-Proposed exit criteria -Contract types the metrics to be 

for the next phase -Proposed exit criteria included as reportable 

-Budgeting for the next phase items in the contract. 

_ -Budgeting _ 

Table 5-3. Sample Learning Objectives. 

This design approach uses generally accepted design principles that are already 
included as a feature in several authoring tools. For example, the following design 
approach includes the need for interactivity and user friendly navigation. 
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Module 1 

Acquisition Strategy 
(MDAP) 

Module 2 

Acquisition Strategy 
(MAIS) 

Module 7 

Risk Management 

Module 8 
Analysis and 
Control 

Design Approach: 

(a) Make hyperlinks 
available on the first 
page along with 
instructions for 
completing the module. 
Links correspond to the 
read-ahead (documents 
describing the scenario 
for this module, and the 
instructions for fulfilling 
this module’s 
requirements) and the 
Defense Acquisition 
DeskBook Online. 

(b) There are six pages 
with blank window 
backgrounds, each 
displaying 24 evenly 
spaced vertical lines 
delineating the months. 

(c) Movable bar and 
triangle shaped icons 
representing the various 
tasks possible to perform 
dining that phase (e.g. 
conduct Analysis of 
Alternatives, etc.) are 
available at the bottom of 
the window for students 
to "drag and drop" in the 
position predetermined 
by their review of the 
scenario. 

Design Approach: 

(a) Make hyperlinks 
available on the first page 
along with instructions for 
completing the module. 
Links correspond to the 
read-ahead (documents 
describing the scenario for 
this module, and the 
instructions for fulfilling 
this module’s 
requirements) and the 
Defense Acquisition 
DeskBook Online. 

(b) There are six pages 
with blank window 
backgrounds, each 
displaying 24 evenly 
spaced vertical lines 
delineating the months. 

(c) Movable bar and 
triangle shaped icons 
representing the various 
tasks possible to perform 
during that phase (e.g. 
conduct Analysis of 
Alternatives, etc.) are 
available at the bottom of 
the window for students to 
"drag and drop" in the 
position predetermined by 
their review of the 
scenario. 

Design Approach: 

(a) Make hyperlinks 
available on the first 
page along with 
instructions for 
completing the module. 
Links correspond to the 
read-ahead (documents 
describing the scenario 
for this module, and the 
instructions for fulfilling 
this module's 
requirements) and the 
Defense Acquisition 
DeskBook Online. 

(b) The next page 
provides a table with text 
and number capture 
windows that students 
fill in with event name, 
probability, and impact 
of occurrence. 

(c) The next page(s) 
presents the user with a 
calculated risk exposure 

; figure and requires 
! students to prioritize risk 
' reduction measures 
based on a cost benefit 
analysis. 

Design Approach: 

(a) Make hyperlinks 
available on die first 
page along with 
instructions for 
completing the module. 
Links correspond to the 
read-ahead (documents 
describing the scenario 
for this module, and the 
instructions for 
fulfilling this module's 
requirements) and the 
Defense Acquisition 
DeskBook Online. 

(b) The next several 
pages allow students to 
input the raw data 
gleaned from the 
scenario for the relevant 
metrics identified 
during Module 7. 

(c) The next page 
provides a display using 
graphs, charts or other 
appropriate visual 

| display of the data 
entered. 


Table 5-4. Design Approach. 
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Module 1 
Acquisition 
Strategy (MDAP) 

Module 2 

Acquisition Strategy 
(MAIS) 

Module 7 

Risk Management 

Module 8 

Analysis and Control 

(d) The next page 
prompts the user to 
enter a written 
summary that 
characterizes their 
overall acquisition 
strategy. 

(e) Follow on pages 
require the students to 
answer questions using 
text capture boxes 
addressing their 
approach to selecting 
contract types, CAIV 
considerations, 
budgeting breakdown, 
and IPT 

implementation as they 
relate to an MDAP 
program. 

(d) The next page 
prompts the user to enter 
a written summary that 
characterizes their 
overall acquisition 
strategy and how that 
strategy differs from an 
MDAP strategy. 

(e) Follow on pages 
require the students to 
answer questions using 
text capture boxes 
addressing their 
approach to selecting 
contract types, CAIV 
considerations, 
budgeting breakdown, 
and IPT implementation 
as they relate to an 

MAIS program. 

(d) Follow-on 
page(s) query 
students for metrics 
to include as 
reportable items in 
the contract to 
monitor risk factors. 

(e) Follow on pages 
require the students 
to answer questions 
using text capture 
boxes addressing 
their overall risk 
Management 

Strategy. 

(d) The next page(s) 
provides boxes arranged 
in an ordinal sequence 
to allow students to 
express their qualitative 
opinion about the status 
of a program via 
checkmarks. 

(e) Follow on page(s) 
use text capture boxes 
requiring the students to 
provide assessment of 
overall program status 
as they interpret it from 
the displayed data and 
to make any 
recommendation for 
senior management 
concerning necessary 
corrective actions. 


Table 5-4. Design Approach (continued). 


2. Layout 

Having demonstrated a method of conducting the planning phase, the next step is 
to use one of the preceding module plans and develop an ATAPMS module using a 
commercial authoring tool. This section illustrates how an ATAPMS module would look 
from the developer and the user perspective. 

The authoring tool used to demonstrate this off-core study is "ToolBook II, 
Assistant 7" distributed by Asymetrix Learning Systems. While there are commercially 
available authoring tools, such as MacroMedia's "Authorware", that contain the 
functionality required for an ATAPMS, ToolBook 13 is not among them. ToolBook II 
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does not possess the text capture and retrieval functionality necessary to facilitate a 
qualitative review of the user's input. Therefore, ToolBook II is used in this section only 
to generate the figures needed to illustrate the user's view of a module that appears to 
contain the functionality desirable for an ATAPMS. 

a) Module Posting 

Once the module is created, ToolBook II provides the option to "Import to 
HTML": a feature that automatically packages the module in a directory ready for 
publication on a website (Figure 5-1). Within each directory are all of the files necessary 
for posting the module on a web server (Figure 5-2). Because all of these files are 
automatically generated by the authoring tool with the execution of one command, an 
instructor acting as developer only needs to be proficient with developing the actual 
content of the module, and not the secondary task of preparing the module to be accessed 
from a web server. 
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Figure 5-1. Web Enabled Directories. 
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Figure 5-2. Inside Module 8’s Directory. 
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b) Module Presentation 

After loading the module directories onto a web server, a simple web page 
is all that is needed to provide a link to each of the scenarios and module directories. The 
instructor can use this web site as a "home page" for the courseware, allowing students 
convenient access to the scenarios and modules (Figure 5-3). Students can access this site 
from any location on the Internet well in advance in order to prepare for the scheduled 
lab. 

This schema supports the plug and play concept by using an easily 
reconfigurable interface between the application modules and the students. Any changes 
made by the instructor or application manager to the modules, their directory structures, 
or the scenarios, are transparent to the users. Any additions made to the courseware can 
be easily and quickly accommodated by the web interface. 

c) Module Example 

The following module example demonstrates one method of designing a 
desirable learning experience for a user based on the preceding "Sample Learning 
Objectives" and "Design Approach" presented in Table 5-3 and 5-4 for "Module 8: 
Analysis and Control". Prior to downloading the module, the user would first download 
and review a case scenario (e.g. "Scenario 8" shown in Figure 5-3) that is also linked to 
course home page. Once familiar with the scenario, the user can return to the home page 
and download the actual module (e.g. "Lab 8 Module" shown in Figure 5-3). 
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Figure 5-3. Web Site Used as a "Home Page." 


Figure 5-4 provides an example module front page as viewed by the user 
after downloading it from the home page. In this example, the module fosters an 
integrative capstone approach to the lesson and therefore is designed be used by student 
teams. Accordingly, the users would enter their "team identification" so the instructor 
could distinguish between the various teams' module outputs. There is also an 
administrative information and HTTP hyper-link to the scenario read-ahead and the 
Department of Defense Acquisition DeskBook web site. 
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Figure 5-4. Module Front Page. 

The next page provides an example of requiring students to extract the 
appropriate information to demonstrate the understanding of a scenario. The module page 
presents a query to the user. The previously viewed scenario concerning the application of 
software metrics provided the user with the quantitative software metrics data required to 
answer the query. Figure 5-5 shows the student’s input to those queries based on their 
understanding of the scenario. Although it is not shown in this example, there would be 
as many subsequent pages as necessary to capture the relevant metrics. 
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Figure 5-5. Numerical Input. 


After requiring the students to demonstrate an ability to distill factual 
material from the given scenario, the following pages prompt them to provide qualitative 
interpretations of the factual data. The next few pages of the module query the students 
for their assessment of the program's status based only on the numerical data (Figure 5-6). 


The students are prompted to evaluate the status of the various components (e.g. 
"requirements" or "design") of the scenario program based only on the reading of the 
scenario. In this example, the students will check the block to provide their assessment. 


Again, although it is not shown in this example, there would be as many subsequent 
pages as necessary to capture the assessment of all the desired areas. 
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After the students complete their assessment, the follow-on page of the 
module then provides a graphic that transforms the numerical data into a visual display 
(Figure 5-7). Next, to help illustrate the usefulness of comprehensive graphics, the 
module allows the students to reevaluate the previously answered questions (Figure 5-8). 
The intent of this comparison is to measure the difference in the student's perception of 
the data based on numerical versus a visual reference and demonstrate the advantage of a 
graphical presentation for managers. 
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Figure 5-6. Query Based on Numerical Data. 
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Figure 5-7 From [Ref. 45]. Visual Display of Da ta. 



Link to Case Study Read-ahead 


Link to DoD DeskBook 


Figure 5-8. Query Based on Visual Display. 
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The final pages of this module require the students to elaborate on their 
overall assessment of the program status. Figure 5-9 requires the students to type their 
evaluation of the program's "health" into a text capture box. Figure 5-10 requires the 
students to type into a text capture box their recommendations for corrective actions. This 
feedback allows an instructor to measure the student's comprehension of a program’s 
status. 
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Figure 5-10. Recommendation for Corrective Action. 

d) Module Output 

The format of a module's output varies with the different commercial 
authoring tools. A review of several of the tools available shows that the feedback options 
have a wide range. Some authoring tools provide an instructor with only a simple 
numerical score based on the user's performance. Others offer the instructor the capability 
to view the user's score and the specific answers chosen for the questions presented. The 
most useful authoring tools, and the only category that meets the requirements for an 
ATAPMS, provide the capability to export all the information entered by the user. 

The ability to capture and retrieve all of the user's input is essential. This 
feature is necessary to allow an instructor to perform a qualitative review of essay type 
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answers. The capture and retrieval capability is also necessary for writing numerical input 
to a database and then developing dynamic graphics from that information. 

D. CHAPTER SUMMARY 

The most practical option for building the ATAPMS modules is through the use 
of commercial based authoring tools. These tools have many advantages over developing 
new code: developing new modules or adjusting existing modules is easier and costs less, 
requires less time, and are easier to maintain. Even though authoring tools are the most 
practical option, care must be taken to select a tool that provides the applicable features 
required by an ATAPMS such as the capability to capture and retrieve the content of a 
user's session. 
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VI. CONCLUSION 


A. CONCLUSIONS 

The DoD has identified the education and training of the acquisition workforce as 
a strategy to help make the acquisition system more effective and efficient. As a result, 
the DOD established the DAU to provide the required education and training. More 
recently, EO 13111 and the DRI have presented a mandate for the DoD to find ways to 
use technology to further this goal. In concert with this mandate, the DoD's ADL 
initiative will provide the added reach and the decreased cost of education and training 
through the creation of industry-wide standards for educational software. 

Even though the DAU is responding to this mandate and using technology to 
provide interactive distance learning courses, there is one type of course not yet offered. 
There are currently no automated courses that include a feature desired by several 
instructors at the NPS: the capability to allow instructors to qualitatively assess a student's 
work. 

B. RECOMMENDATIONS 

To help satisfy this need, this thesis developed the top-level requirements and 
illustrated a notional Acquisition Program Management course for an ATAPMS model 
tailored to the needs of instructors at the NPS. Most importantly, ATAPMS needs to 
accommodate many interchangeable modules and must scale to the user’s needs. 
Additionally, the program must also incorporate the Advanced Distance Learning 
precepts. Table 6-1 provides a comprehensive summary of the recommendations for the 
structure, strategies, and topics for the application of an ATAPMS at the NPS. 
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Structure 

Content 

Strategies 

-Use a capstone style approach 
to the content 
-Accommodate at least 10 
weekly topic modules 
-Be capable of presenting well 
developed thought- 
provoking scenarios and 
questions 

-Ensure modular architecture 
for scalability and a plug and 
play capability 
-Be IMS compliant 
-Ensure Accessibility 
-Ensure Durability 
-Ensure Re-usability 
-Ensure Cost effectiveness 

-Acquisition Strategy for a Major Defense 
Acquisition Program (MDAP) 

-Acquisition Strategy for a Major 

Automation Information System (MAIS) 
-System Engineering Process 
-Defining Critical Design & Capabilities 
-Analysis & Control 
-Risk Management 
-Acquisition Logistics 
-Contracting & Contract Management 
-Budgeting 
-Test & Evaluation 
-Production & Fielding 
-Work Breakdown Structure & Earned Value 
-Software Management 
-People Factors 

-Chronological by Phase 
-Concept Centric 
-Systems Engineering 
Centric 

-Logistics Centric 
-Hybrid Approach 


Table 6-1. Summary Table. 


The most practical option for building the ATAPMS modules is through the use 
of commercial based authoring tools. However, care must be taken to select a tool that 
provides the applicable features required by an ATAPMS as listed in the "Structure" 
column in table 6-1. Additionally, any tool chosen to build ATAPMS modules must be 
capable of capturing and retrieving the content of a user’s session to enable the dynamic 
generation of graphics and an instructors qualitative assessment of a student's work. 

C. RECOMMENDATIONS FOR FURTHER RESEARCH 

There are three topics that require further research. The first topic is the 
comparison of authoring tools to determine which tool best matches the requirements of 
the instructors at the NPS as well as the DAU. The second related, but distinct subject, is 
evaluating the need for a system to provide management for the users and the ATAPMS 
modules. The third and final topic involves developing a working prototype for a course. 

While there are several authoring tools available, a careful review is required to 
find an appropriate match for the individual consortium school. Because it is possible to 
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find more than one product that meet the requirements outlined in this thesis, further 
research is needed to evaluate the candidate tools' strengths and weaknesses against each 
other in conjunction with the needs of the DAU consortium school. 

In addition to an authoring tool, further research should also include the selection 
of a management tool. Management tools are used for the "command and control" of 
courses and students. They provide a method of centrally controlling the various learning 
activities, including course delivery, access, collaboration, and performance tracking. 

Most authoring tools can operate independent of their associated management 
applications. However, the majority of tools encountered during the research of this thesis 
had a corresponding management application that was more expensive than its associated 
authoring tool. For example, one vendor only charged $1,200.00 for the authoring 
software, but charged a minimum of $8,000 for the use of the management tool. 
Therefore, the utility of incurring an additional expense to procure a management tool 
requires additional analysis. 

Finally, once the questions surfaced by the aforementioned topics have been 
resolved, the next step is to develop a working prototype for an entire course offering. 
The development of this prototype has two requirements. The first task is to gain 
familiarity of the features offered by the selected authoring and management tools. The 
second task is to work closely with the cognizant academic department to develop a 
relevant lesson plan that can be supported by the authoring and management tools. 

Because these topics require careful analysis before implementing a workable 
ATAPMS, they are ideal subjects for further research. 
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APPENDIX A. SELECTING AN APPROPRIATE AUTHORING TOOL 

There are currently over 90 different authoring tools for designing and 
implementing a computer based training course. Selecting the most appropriate tool 
without using a systematic procedure is a difficult task at best. One methodical approach 
to this task involves a thorough analysis of the user’s needs and then locating the product 
that best matches them. 

To assist a user assess his or her needs, most on line learning sources provide a 
list of questions to help illustrate the features available for authoring tools. To help match 
the overall needs of the user, the enclosed tables consolidate a comprehensive list of 
features characteristics available in authoring tools. 

There are several resources available through the World Wide Web that provide a 
listing of tools and their features. For example, there is a website at 
[http://www.umr.edu/~rhall/research/webleaming/products.html] that lists links to over 
20 websites related to authoring tools and products and an additional 10 websites related 
to the assessment of authoring tools. 

Once the user has determined his or her overall need and identified the required 
features listed on the following tables, the next step is to find the authoring tool that best 
matches the desired features listed. One site, shown at Figure A-l and located at 
[http://www.ctt.bc.ca/landonline/option2.html], allows a user to perform an online 
evaluation of up to 33 authoring tools within its database. Using this online evaluation 
tool, the user can select the applicable features and the evaluation tool will display the 
application name of the authoring tool(s) that most closely match the features indicated. 
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The other option for this program is to enter the authoring tool name and the evaluation 
tool will display checkmarks in the corresponding feature boxes. 

As an additional resource, an online glossary of terms is available from the same 
provider at [http://www.ctt.bc.ca/landonline/glossary.html]. 
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Figure A-l From [Ref. 46]. An Online Authoring Tool Comparison Program 
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After selecting one or more candidate tools, the next step is to contact the vendor 
and verify the information concerning the advertised features and their actual cost 
structure. Usually, the vendor employs a sales staff to field the rudimentary questions. 
However, it is helpful to obtain a contact number to the technical staff for questions that 
require a more detailed response. 

In addition to gathering cost and feature data, testing an online demonstration or 
obtaining a trial version of the candidate’s software is the next important step. Most 
vendors post an online course that demonstrates the capabilities of its authoring tool. 
Navigating through the “demo’s” offered through the various candidates’ websites is an 
excellent way to compare the each vendor’s product. Ultimately, obtaining a trial version 
of the actual authoring tool is the ideal method for familiarizing oneself with the all the 
tools features and capabilities. 

Listed below is a comprehensive set of questions to help illustrate the features 
available for authoring tools. These questions are derived from a site 
[http://multimedia.marshall.edU/cit/webct/compare/comparison.html#develop] that 
presents a comparison of online course delivery products. However, because the specific 
products and their versions listed on the site are subject to frequent changes, this 
appendix presents only the questions listed on the site and not their results. 
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ADMINISTRATIVE FEATURES 

University will have sole ownership of custom code used to create courses 

University will have sole ownership of course content _ 

Platform provider will provide technical support to University students 

Platform provider will provide technical support to University faculty and staff 

Platform provider will host courses on their server 

Platform provider will advertise courses 

Platform provider can provide documentation and contacts to demonstrate a positive track 
record with higher education 

Pricing structure is based upon number of students within the course 

Platform is focused on locally developed courses as opposed to "canned" courses 

Platform has large startup cost with minimal continual costs 

Table A-l. Administrative Features. 

ADMINISTRATOR TOOLS 

Server 

Client/Web interface 

Authorization tools 

Logout feature 

Resource monitoring 

Remote access tools 

Crash recovery tools 

Student support tools 

Instructor support tools 

Administrator support tools 

Built-in file management tools 

Ability to export raw data 

Customization of text messages 

Resume session function 

Security access 

Variable level of security 

Online registration 

Registered markers 

Batch process for inputting student accounts 

Guest account creation 


Table A-2. Administrative Tools. 
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DEVELOPMENTAL FEATURES _ 

Content format will allow for simple transfer to/ffom another vendor's platform _ 


Platform uses open data standard so that it can communicate with existing university 
database applications _ 


Content can be authored on PCs running Windows 95/98/NT _ 

Content can be authored on Macs running OS 7.5 or greater _ 

Courses can be taken using a PC running Windows 95/98/NT_ 


Courses can be taken using a Macintosh running OS 7.5 or greater _ 

Platform provider is supportive of implementing IMS standard within product_ 


Platform provider is supportive of implementing AICC standards within product 


Platform utilizes standard HTML for content creation _ 

Platform is structured so students can view all of their current courses when they log on 


Platform's server software will run on DEC Unix _ 

Platform's server software will run on Windows NT _ 


Multiple choice questions can be created\scored with platform’s authoring software 


True\False questions can be created\scored with platform's authoring software _ 

Matching questions can be created\scored with platform's authoring software 


Short answer questions can be created\scored with platform's authoring software _ 

Essay questions can be created\scored with platform’s authoring software_ 


Platform supports question database for management of test questions _ 


Platforms supports reporting features for test questions _ 

Platform supports Microsoft Internet Explorer 4.x and newer browsers 


Platform supports testing stage for courses to debugged before making them live to 

students __ 

Platform allows author to view course as student without logging out 


Platform has built-in threaded discussion list capabilities _ 

Platform has built-in chat capabilities _ 


Platform can be integrated with Real networks video and audio products 


Platform can be integrated with Macromedia Shockwave products _ 

Vendor provides development services_ 


Management component will create reports for tracking student progress _ 

Platform has a feature to import existing test questions in a tab-delimited format 


Table A-3. Developmental Features. 



HARDWARE REQUIREMENTS _ 

UNIX server _ 

NT 4.0 server _ 

CGI-enabled Web server _ 

Java-enabled Web browser _ 

Mac OS _ 

Solaris _ 

Linux _ 

Table A-4. Hardware Requirements 
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INSTRUCTIONAL FEATURES 

Platform choice can be customized to incorporate SPU distinctives 

Faculty to student asynchronous communication is possible 

Faculty to student synchronous communication is possible 

Faculty can make their own changes to content 

Training is provided for faculty 

Courses can have consistent interface 

Platform supplies access to library resources other than the university's present holdings 

Online help is available to help student use library resources 

Platform includes an internal e-mail client 

Platform has e-mail management capabilities for students 

Platform has e-mail management capabilities for faculty 

Platform supports multiple instructors for a single course 

Table A-5. Instructional Features. 

INSTRUCTOR TOOLS 

Course planning 

Course managing 

Fast course revising 

Course monitoring 

Instructional designing 

Presenting information 

On-line testing 

On-line grading 

Managing records 

No HTML knowledge required 

Customization of student curriculum 

Student tracking 

Automated grading 

Level of control over design 

Instructor can assign specific course material to individual or group of students 

Multiple choice self test tutorial questions - (automatic marking) 

"Fill in the blank" self test tutorial questions - (automatic marking) 

Customized feedback to tutorial questions 

Redirect path of tutorial depending on question answers 

Timed quizzes (graded with permanent mark retention) 

On line marking and grades management of timed quizzes 

Generate random set of questions 

Allows developer to preview course as a student 


Table A-6. Instructor Tools. 
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SOFTWARE COSTS 

Start-up costs _ 

On-going costs _ 

Site pricing _ 


Table A-7. Software Costs. 


STUDENT TOOLS 

Authentication 

Bookmark management 

Multimedia support 

Private e-mail 

File submissions 

Threaded discussions 

Course Chat rooms 

Logged chat 

Whiteboard 

Self-assessing 


Desktop based file management for uploading to server 


Un-timed quizzes 

One question-at-a-time function 

Bulletin board/conferencing tools 

Image database 

Student access to own grades 

Access to course grade distribution 

Automated glossary tool 

Automated index tool 

Online assistance 

Search tool for course content 

Student presentations area 

Allows students to view all current courses in which they are registered after logging in 


Table A-8. Student Tools. 


TECHNICAL SUPPORT _ 

External e-mail _ 

Security features _ 

Assignable administrator role _ 

Batch add instructors _ 

Batch add students __ 

Template creations tools _ 

Built-in instructor manual __ 

Built-in student manual _ 

Database _ 

Table A-9. Technical Support. 
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